Sep 16 18:38:10.751446 ip-10-0-139-72 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:38:10.751459 ip-10-0-139-72 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:38:10.751468 ip-10-0-139-72 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:38:10.751675 ip-10-0-139-72 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:38:22.087357 ip-10-0-139-72 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:38:22.087375 ip-10-0-139-72 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1d6639f5c7454719aaa4b25dd194dcb0 --
Sep 16 18:40:03.986475 ip-10-0-139-72 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:40:04.383102 ip-10-0-139-72 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:04.383102 ip-10-0-139-72 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:40:04.383102 ip-10-0-139-72 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:04.383102 ip-10-0-139-72 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:40:04.383102 ip-10-0-139-72 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:04.384909 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.384805    2552 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:40:04.389013 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.388985    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:04.389013 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389012    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:04.389013 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389016    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389021    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389025    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389028    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389031    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389034    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389038    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389040    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389043    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389046    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389049    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389052    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389055    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389057    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389060    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389063    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389066    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389069    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389072    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389083    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:04.389153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389098    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389102    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389105    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389107    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389110    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389113    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389116    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389119    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389121    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389124    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389126    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389129    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389132    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389135    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389139    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389144    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389148    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389152    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389155    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389158    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:04.389652 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389161    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389165    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389168    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389171    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389174    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389177    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389180    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389182    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389185    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389187    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389192    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389195    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389198    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389201    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389205    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389208    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389211    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389214    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389216    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389220    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:04.390222 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389223    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389225    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389228    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389232    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389235    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389237    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389240    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389243    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389246    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389253    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389256    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389259    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389264    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389266    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389269    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389272    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389275    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389278    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389280    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:04.390729 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389283    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389286    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389288    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389291    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389294    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389296    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389299    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389302    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389305    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389307    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389310    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389313    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389316    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389318    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389320    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389323    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389326    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.389329    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390131    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390139    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:04.391223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390142    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390145    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390148    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390151    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390155    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390158    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390161    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390163    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390166    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390169    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390171    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390174    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390177    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390180    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390182    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390185    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390187    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390190    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390194    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:04.391714 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390198    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390201    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390204    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390207    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390210    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390213    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390216    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390218    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390222    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390226    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390230    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390233    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390236    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390238    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390241    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390243    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390246    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390249    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390252    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:04.392233 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390255    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390257    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390259    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390262    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390265    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390267    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390272    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390275    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390277    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390280    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390282    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390285    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390288    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390291    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390294    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390297    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390300    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390303    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390305    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390308    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:04.392753 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390310    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390313    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390315    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390318    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390321    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390323    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390326    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390328    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390331    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390333    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390336    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390339    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390342    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390345    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390347    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390350    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390352    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390355    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390357    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390360    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:04.393284 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390362    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390365    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390368    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390372    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390376    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390379    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390382    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390386    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390389    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390392    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390395    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390398    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390401    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390403    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390406    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390409    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390412    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390414    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390417    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390838    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:04.393793 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390844    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390846    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390849    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390852    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390855    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390857    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390860    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390862    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390866    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390868    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390871    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390873    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390876    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390878    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390881    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390883    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390886    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390889    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390892    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390895    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:04.394321 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390898    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390901    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390904    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390906    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390909    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390912    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390915    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390917    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390920    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390922    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390926    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390930    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390933    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390936    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390939    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390941    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390945    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390947    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390950    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:04.394836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390953    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390955    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390958    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390960    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390963    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390965    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390968    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390970    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390973    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390975    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390982    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390984    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390987    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390990    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390993    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390996    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.390998    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391001    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391003    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391006    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:04.395329 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391009    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391013    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391016    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391018    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391021    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391023    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391026    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391028    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391032    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391036    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391039    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391043    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391045    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391048    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391052    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391054    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391057    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391060    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391063    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:04.395826 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391066    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391069    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391072    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391076    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391079    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391097    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391100    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391103    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391106    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391109    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391112    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391115    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391118    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391120    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391123    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391126    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391128    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391131    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391134    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391136    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:04.396340 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391231    2552 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391272    2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391280    2552 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391284    2552 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391290    2552 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391293    2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391298    2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391303    2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391307    2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391310    2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391314    2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391318    2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391321    2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391325    2552 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391328    2552 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391333    2552 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391335    2552 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391340    2552 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391345    2552 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391348    2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391352    2552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391355    2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391358    2552 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391362    2552 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:40:04.396838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391365    2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391369    2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391372    2552 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391375    2552 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391378    2552 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391381    2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391384    2552 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391387    2552 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391391    2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391395    2552 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391397    2552 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391400    2552 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391403    2552 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391407    2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391412    2552 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391415    2552 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391418    2552 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391421    2552 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391425    2552 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391429    2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391431    2552 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391435    2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391438    2552 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391442    2552 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391446    2552 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:40:04.397441 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391450    2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391453    2552 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391456    2552 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391459    2552 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391462    2552 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391466    2552 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391469    2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391473    2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391476    2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391479    2552 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391482    2552 flags.go:64] FLAG: --help="false"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391485    2552 flags.go:64] FLAG: --hostname-override="ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391489    2552 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391492    2552 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391495    2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391498    2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391501    2552 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391504    2552 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391507    2552 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391510    2552 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391513    2552 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391517    2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391520    2552 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391523    2552 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:40:04.398137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391526    2552 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391529    2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391532    2552 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391535    2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391538    2552 flags.go:64] FLAG: --lock-file=""
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391540    2552 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391545    2552 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391548    2552 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391556    2552 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391559    2552 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391561    2552 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391564    2552 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391567    2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391571    2552 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391574    2552 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391577    2552 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391582    2552 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391584    2552 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391589    2552 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391592    2552 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391595    2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391598    2552 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391601    2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391604    2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391607    2552 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:40:04.398860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391610    2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391619    2552 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391622    2552 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391626    2552 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391629    2552 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391632    2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391639    2552 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391642    2552 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391645    2552 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391649    2552 flags.go:64] FLAG: --port="10250"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391651    2552 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391654    2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0aa82ec696a44b538"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391657    2552 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391660    2552 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391665    2552 flags.go:64] FLAG: --register-node="true"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391669    2552 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391673    2552 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391676    2552 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391679    2552 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391682    2552 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391686    2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391690    2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391693    2552 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391696    2552 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391699    2552 flags.go:64] FLAG: --runonce="false"
Sep 16 18:40:04.399600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391702    2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391705    2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391709    2552 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391712    2552 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391715    2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391718    2552 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391722    2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391725    2552 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391729    2552 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391732    2552 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391735    2552 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391738    2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391741    2552 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391745    2552 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391748    2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391754    2552 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391757    2552 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391760    2552 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391764    2552 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391767    2552 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391770    2552 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391773    2552 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391778    2552 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391781    2552 flags.go:64] FLAG: --v="2"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391785    2552 flags.go:64] FLAG: --version="false"
Sep 16 18:40:04.400291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391789    2552 flags.go:64] FLAG: --vmodule=""
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391794    2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.391797    2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391941    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391945    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391949    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391952    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391955    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391958    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391960    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391963    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391966    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391969    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391972    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391974    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391977    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391980    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391983    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391985    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391988    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:04.400938 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391991    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391994    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.391997    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392000    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392003    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392005    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392008    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392011    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392013    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392018    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392020    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392023    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392026    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392030    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392034    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392037    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392040    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392043    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392046    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:04.401457 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392049    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392052    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392054    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392057    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392059    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392062    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392065    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392067    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392070    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392072    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392077    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392080    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392083    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392099    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392102    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392106    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392109    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392111    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392114    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:04.401979 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392117    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392120    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392123    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392128    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392130    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392133    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392136    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392138    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392141    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392144    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392146    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392158    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392161    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392164    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392167    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392169    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392172    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392175    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392177    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392181    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:04.402485 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392183    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392186    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392189    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392191    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392194    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392197    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392201    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392204    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392207    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392211    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392214    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392217    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392220    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392222    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392225    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392229    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392231    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392234    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392237    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392240    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:04.403057 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392243    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:04.403595 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392245    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:04.403595 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392249    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:04.403595 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.392251    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:04.403595 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.392784    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:04.403595 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.400519    2552 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:40:04.403595 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.400542    2552 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:40:04.403595 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.401267    2552 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:40:04.403595 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.401278    2552 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:40:04.403595 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401346    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:04.403595 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401352    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:04.403595 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401355    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:04.403595 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401359    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:04.403595 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401363    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:04.403595 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401366    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:04.403595 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401369    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401372    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401375    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401377    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401380    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401382    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401385    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401389    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401391    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401394    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401397    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401400    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401402    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401405    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401407    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401410    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401413    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401415    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401418    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401421    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:04.404155 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401423    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401427    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401429    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401435    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401438    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401440    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401443    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401446    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401448    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401451    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401453    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401456    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401458    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401461    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401464    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401467    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401469    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401472    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401475    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401477    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:04.404942 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401480    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401483    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401485    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401488    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401490    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401493    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401496    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401499    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401501    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401504    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401506    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401510    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401515    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401517    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401521    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401524    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401528    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401531    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401534    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401537    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:04.405599 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401540    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401543    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401546    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401548    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401551    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401553    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401556    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401558    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401561    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401563    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401566    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401568    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401571    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401574    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401577    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401580    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401584    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401587    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:04.406216 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401590    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:04.406715 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401593    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:04.406715 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401597    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:04.406715 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401602    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:04.406715 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401605    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:04.406715 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401607    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:04.406715 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401611    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:04.406715 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401613    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:04.406715 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401616    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:04.406715 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401620    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:04.406715 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401622    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:04.406715 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401626    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:04.406715 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401629    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:04.406715 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401631    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:04.406715 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401634    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:04.406715 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.401639    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401796    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401802    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401805    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401809    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401812    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401815    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401819    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401822    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401824    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401827    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401830    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401832    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401835    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401838    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401840    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401843    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401846    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401848    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401851    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:04.407106 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401854    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401857    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401859    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401862    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401865    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401867    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401871    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401873    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401876    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401879    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401882    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401885    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401887    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401890    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401892    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401895    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401898    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401900    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401903    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:04.407606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401905    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401909    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401914    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401917    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401920    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401922    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401925    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401928    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401931    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401934    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401937    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401939    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401942    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401945    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401948    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401950    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401953    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401955    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401958    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:04.408074 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401961    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401964    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401967    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401970    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401973    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401975    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401978    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401981    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401983    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401987    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401989    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401992    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401995    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.401998    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402001    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402005    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402008    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402010    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402013    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402016    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:04.408584 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402018    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402021    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402023    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402027    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402030    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402034    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402037    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402040    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402042    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402045    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402048    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402050    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402058    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402060    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402063    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402066    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402068    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402072    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402074    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402077    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:04.409103 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402079    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:04.409618 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:04.402082    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:04.409618 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.402101    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:04.409618 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.402240    2552 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:40:04.409618 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.405734    2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:40:04.409618 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.407113    2552 server.go:1019] "Starting client certificate rotation"
Sep 16 18:40:04.409618 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.407220    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:04.409618 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.407273    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:04.410312 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.410291    2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:04.415754 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.415719    2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:04.427797 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.427770    2552 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:40:04.431097 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.431068    2552 log.go:25] "Validated CRI v1 image API"
Sep 16 18:40:04.432165 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.432129    2552 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:40:04.437566 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.437541    2552 fs.go:135] Filesystem UUIDs: map[41f3daa2-4143-48fe-9f25-7b1ac519ad47:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ea419270-e31e-49af-a924-ade44ed700c8:/dev/nvme0n1p3]
Sep 16 18:40:04.437669 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.437564    2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:40:04.438360 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.438336    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:04.445028 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.444859    2552 manager.go:216] Machine: {Timestamp:2026-09-16 18:40:04.442321763 +0000 UTC m=+0.336509023 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099932 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a1c2ddc8abeeef3d9d5652feae92f SystemUUID:ec2a1c2d-dc8a-beee-f3d9-d5652feae92f BootID:1d6639f5-c745-4719-aaa4-b25dd194dcb0 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ee:f3:4b:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ee:f3:4b:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:b4:ed:17:83:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:40:04.445028 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.445009    2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:40:04.445211 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.445135    2552 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:40:04.448960 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.448910    2552 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:40:04.449201 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.448969    2552 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-72.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:40:04.449880 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.449863    2552 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:40:04.449914 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.449883    2552 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:40:04.449914 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.449902    2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:04.450567 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.450548    2552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:04.451567 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.451549    2552 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:40:04.454969 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.454949    2552 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:04.455167 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.455155    2552 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:40:04.455213 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.455196    2552 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:40:04.455213 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.455210    2552 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:40:04.455270 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.455230    2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:40:04.455270 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.455246    2552 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:40:04.455270 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.455259    2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:40:04.455893 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.455867    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:04.455987 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.455900    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:04.459038 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.459007    2552 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:40:04.461352 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.461329    2552 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:40:04.461461 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.461365    2552 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:40:04.463082 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.463068    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:40:04.463156 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.463119    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:40:04.463156 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.463129    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:40:04.463156 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.463138    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:40:04.463156 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.463148    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:40:04.463156 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.463153    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:40:04.463290 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.463167    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:40:04.463290 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.463173    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:40:04.463290 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.463184    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:40:04.463290 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.463189    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:40:04.463290 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.463199    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:40:04.463290 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.463204    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:40:04.465353 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.465142    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:40:04.465353 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.465181    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:40:04.469169 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.469130    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-72.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:40:04.469330 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.469305    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:40:04.469381 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.469353    2552 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-72.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:40:04.470217 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.470202    2552 server.go:1268] "Started kubelet"
Sep 16 18:40:04.470360 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.470301    2552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:40:04.470416 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.470390    2552 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:40:04.470416 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.470311    2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:40:04.471226 ip-10-0-139-72 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:40:04.472565 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.472547    2552 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:40:04.472725 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.472712    2552 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:40:04.474719 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.474681    2552 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:40:04.475875 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.474852    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-72.ec2.internal.18d5e12ba22a5db6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-72.ec2.internal,UID:ip-10-0-139-72.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-72.ec2.internal,},FirstTimestamp:2026-09-16 18:40:04.470169014 +0000 UTC m=+0.364356273,LastTimestamp:2026-09-16 18:40:04.470169014 +0000 UTC m=+0.364356273,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-72.ec2.internal,}"
Sep 16 18:40:04.476324 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.476303    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:04.476916 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.476883    2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:40:04.477174 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.477156    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-72.ec2.internal\" not found"
Sep 16 18:40:04.477507 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.477465    2552 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:40:04.477507 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.477468    2552 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:40:04.477507 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.477488    2552 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:40:04.477694 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.477588    2552 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:40:04.477694 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.477596    2552 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:40:04.477867 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.477854    2552 factory.go:55] Registering systemd factory
Sep 16 18:40:04.477911 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.477872    2552 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:40:04.478080 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.478067    2552 factory.go:153] Registering CRI-O factory
Sep 16 18:40:04.478131 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.478082    2552 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:40:04.478160 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.478140    2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:40:04.478160 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.478154    2552 factory.go:103] Registering Raw factory
Sep 16 18:40:04.478212 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.478165    2552 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:40:04.478458 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.478449    2552 manager.go:349] Starting recovery of all containers
Sep 16 18:40:04.489811 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.489599    2552 manager.go:354] Recovery completed
Sep 16 18:40:04.494451 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.494406    2552 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-72.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:40:04.494451 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.494427    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:40:04.496661 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.496637    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:04.499239 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.499212    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:04.499338 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.499251    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:04.499338 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.499262    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:04.499909 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.499889    2552 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:40:04.499909 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.499905    2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:40:04.500073 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.499923    2552 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:04.500073 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.499952    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:04.500839 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.500821    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:04.500920 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.500851    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:04.500920 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.500861    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:04.501411 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.501391    2552 policy_none.go:49] "None policy: Start"
Sep 16 18:40:04.501411 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.501409    2552 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:40:04.501518 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.501423    2552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:40:04.501711 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.501688    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bcrpg"
Sep 16 18:40:04.505105 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.505069    2552 policy_none.go:47] "Start"
Sep 16 18:40:04.507448 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.507427    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bcrpg"
Sep 16 18:40:04.509596 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.509508    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-72.ec2.internal.18d5e12ba3e5ec75  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-72.ec2.internal,UID:ip-10-0-139-72.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-72.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-72.ec2.internal,},FirstTimestamp:2026-09-16 18:40:04.499238005 +0000 UTC m=+0.393425265,LastTimestamp:2026-09-16 18:40:04.499238005 +0000 UTC m=+0.393425265,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-72.ec2.internal,}"
Sep 16 18:40:04.552946 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.549820    2552 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:40:04.552946 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.549865    2552 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:40:04.552946 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.549875    2552 server.go:87] "Starting device plugin registration server"
Sep 16 18:40:04.552946 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.550283    2552 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:40:04.552946 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.550296    2552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:40:04.552946 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.550387    2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:40:04.552946 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.550527    2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:40:04.552946 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.550536    2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:40:04.552946 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.551061    2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:40:04.552946 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.551126    2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-72.ec2.internal\" not found"
Sep 16 18:40:04.617779 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.617737    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:40:04.618980 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.618961    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:40:04.619063 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.619021    2552 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:40:04.619063 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.619046    2552 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:40:04.619173 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.619107    2552 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:40:04.621605 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.621584    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:04.650770 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.650675    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:04.651768 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.651747    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:04.651877 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.651779    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:04.651877 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.651792    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:04.651877 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.651818    2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.659113 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.659069    2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.659237 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.659121    2552 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-72.ec2.internal\": node \"ip-10-0-139-72.ec2.internal\" not found"
Sep 16 18:40:04.680174 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.680133    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-72.ec2.internal\" not found"
Sep 16 18:40:04.719400 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.719359    2552 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-72.ec2.internal"]
Sep 16 18:40:04.719557 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.719457    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:04.720590 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.720571    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:04.720718 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.720601    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:04.720718 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.720613    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:04.722335 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.722318    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:04.722455 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.722438    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.722516 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.722496    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:04.723351 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.723333    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:04.723452 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.723361    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:04.723452 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.723371    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:04.723452 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.723383    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:04.723452 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.723407    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:04.723452 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.723417    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:04.724727 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.724711    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.724793 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.724739    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:04.726408 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.726389    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:04.726565 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.726434    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:04.726565 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.726450    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:04.740849 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.740819    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-72.ec2.internal\" not found" node="ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.745683 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.745662    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-72.ec2.internal\" not found" node="ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.779153 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.779112    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b97e76fa4a597a2b9f96eb244b741d8-config\") pod \"kube-apiserver-proxy-ip-10-0-139-72.ec2.internal\" (UID: \"8b97e76fa4a597a2b9f96eb244b741d8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.779153 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.779146    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b809a87ee2a1bceb0e61eaa5e49cd69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal\" (UID: \"5b809a87ee2a1bceb0e61eaa5e49cd69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.779370 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.779234    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b809a87ee2a1bceb0e61eaa5e49cd69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal\" (UID: \"5b809a87ee2a1bceb0e61eaa5e49cd69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.781233 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.781210    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-72.ec2.internal\" not found"
Sep 16 18:40:04.879477 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.879440    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b97e76fa4a597a2b9f96eb244b741d8-config\") pod \"kube-apiserver-proxy-ip-10-0-139-72.ec2.internal\" (UID: \"8b97e76fa4a597a2b9f96eb244b741d8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.879477 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.879479    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b809a87ee2a1bceb0e61eaa5e49cd69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal\" (UID: \"5b809a87ee2a1bceb0e61eaa5e49cd69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.879686 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.879495    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b809a87ee2a1bceb0e61eaa5e49cd69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal\" (UID: \"5b809a87ee2a1bceb0e61eaa5e49cd69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.879686 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.879577    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b97e76fa4a597a2b9f96eb244b741d8-config\") pod \"kube-apiserver-proxy-ip-10-0-139-72.ec2.internal\" (UID: \"8b97e76fa4a597a2b9f96eb244b741d8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.879686 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.879615    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b809a87ee2a1bceb0e61eaa5e49cd69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal\" (UID: \"5b809a87ee2a1bceb0e61eaa5e49cd69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.879686 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:04.879586    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b809a87ee2a1bceb0e61eaa5e49cd69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal\" (UID: \"5b809a87ee2a1bceb0e61eaa5e49cd69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal"
Sep 16 18:40:04.881559 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.881533    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-72.ec2.internal\" not found"
Sep 16 18:40:04.982463 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:04.982359    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-72.ec2.internal\" not found"
Sep 16 18:40:05.043577 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.043533    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal"
Sep 16 18:40:05.048594 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.048569    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-72.ec2.internal"
Sep 16 18:40:05.082720 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:05.082676    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-72.ec2.internal\" not found"
Sep 16 18:40:05.183238 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:05.183198    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-72.ec2.internal\" not found"
Sep 16 18:40:05.283862 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:05.283767    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-72.ec2.internal\" not found"
Sep 16 18:40:05.384345 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:05.384305    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-72.ec2.internal\" not found"
Sep 16 18:40:05.406508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.406463    2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:40:05.406704 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.406666    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:05.484960 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:05.484797    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-72.ec2.internal\" not found"
Sep 16 18:40:05.512747 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.512673    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:35:05 +0000 UTC" deadline="2028-06-13 08:59:09.60989417 +0000 UTC"
Sep 16 18:40:05.512747 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.512750    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15254h19m4.097154472s"
Sep 16 18:40:05.559358 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.559278    2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:05.562304 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.562276    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:05.563592 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.563567    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:05.563727 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.563601    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:05.563727 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.563616    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:05.585839 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:05.585798    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-72.ec2.internal\" not found"
Sep 16 18:40:05.611706 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:05.611669    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b809a87ee2a1bceb0e61eaa5e49cd69.slice/crio-cbcf9e143a476f4261140eacc1eadfd0cca44ab2052782618e222c349141f12b WatchSource:0}: Error finding container cbcf9e143a476f4261140eacc1eadfd0cca44ab2052782618e222c349141f12b: Status 404 returned error can't find the container with id cbcf9e143a476f4261140eacc1eadfd0cca44ab2052782618e222c349141f12b
Sep 16 18:40:05.612637 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:05.612614    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b97e76fa4a597a2b9f96eb244b741d8.slice/crio-aad0c0bb01a38aed558f074dfa76d7ee48a55ca2b644199c1a397473b8628477 WatchSource:0}: Error finding container aad0c0bb01a38aed558f074dfa76d7ee48a55ca2b644199c1a397473b8628477: Status 404 returned error can't find the container with id aad0c0bb01a38aed558f074dfa76d7ee48a55ca2b644199c1a397473b8628477
Sep 16 18:40:05.618471 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.618451    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:40:05.622608 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.622560    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal" event={"ID":"5b809a87ee2a1bceb0e61eaa5e49cd69","Type":"ContainerStarted","Data":"cbcf9e143a476f4261140eacc1eadfd0cca44ab2052782618e222c349141f12b"}
Sep 16 18:40:05.624098 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.624057    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-72.ec2.internal" event={"ID":"8b97e76fa4a597a2b9f96eb244b741d8","Type":"ContainerStarted","Data":"aad0c0bb01a38aed558f074dfa76d7ee48a55ca2b644199c1a397473b8628477"}
Sep 16 18:40:05.686407 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:05.686363    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-72.ec2.internal\" not found"
Sep 16 18:40:05.731791 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.731756    2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:05.786898 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:05.786862    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-72.ec2.internal\" not found"
Sep 16 18:40:05.879243 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.879218    2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:05.978025 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.977983    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal"
Sep 16 18:40:05.991499 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.991463    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:05.993136 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.992637    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-72.ec2.internal"
Sep 16 18:40:05.998962 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:05.998937    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:06.456468 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.456426    2552 apiserver.go:52] "Watching apiserver"
Sep 16 18:40:06.464655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.464622    2552 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:40:06.465120 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.465071    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-9phx7","kube-system/kube-apiserver-proxy-ip-10-0-139-72.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd","openshift-dns/node-resolver-tw7bx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal","openshift-multus/multus-additional-cni-plugins-rnccg","openshift-multus/network-metrics-daemon-zkgpv","openshift-cluster-node-tuning-operator/tuned-89jvz","openshift-image-registry/node-ca-z4qmj","openshift-multus/multus-zvqnt","openshift-network-diagnostics/network-check-target-qbtmz","openshift-network-operator/iptables-alerter-jzjwf","openshift-ovn-kubernetes/ovnkube-node-cs8zv","kube-system/global-pull-secret-syncer-g5lz9"]
Sep 16 18:40:06.466690 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.466663    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:06.466835 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:06.466769    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:06.467868 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.467841    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:06.468003 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:06.467926    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:06.469200 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.469170    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.470542 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.470517    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tw7bx"
Sep 16 18:40:06.472001 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.471362    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:40:06.472001 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.471429    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:40:06.472001 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.471568    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-87z7n\""
Sep 16 18:40:06.472001 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.471666    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:40:06.472685 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.472376    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.472685 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.472555    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:40:06.473344 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.473114    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ngzcs\""
Sep 16 18:40:06.473344 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.473181    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:40:06.473620 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.473602    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.474634 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.474613    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:06.474969 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.474951    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:06.475178 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.475160    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cd458\""
Sep 16 18:40:06.475437 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.475420    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jzjwf"
Sep 16 18:40:06.476019 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.475998    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:40:06.476199 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.476180    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:40:06.476373 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.476352    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.476565 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.476546    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:40:06.476911 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.476890    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:06.476997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.476982    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:40:06.477271 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.477238    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:40:06.477271 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.477257    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zb5df\""
Sep 16 18:40:06.477597 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.477547    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:06.477884 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.477868    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9phx7"
Sep 16 18:40:06.478680 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.478661    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wvflj\""
Sep 16 18:40:06.478680 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.478673    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:06.478950 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.478934    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:40:06.479169 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.479154    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:40:06.479257 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.479216    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:40:06.479747 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.479440    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:40:06.480178 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.479896    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:40:06.480178 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.479933    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:40:06.480310 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.480200    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-txhpc\""
Sep 16 18:40:06.480310 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.480211    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:40:06.480310 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.480225    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:40:06.480455 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.480400    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t7xrx\""
Sep 16 18:40:06.480455 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.480416    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:40:06.482339 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.482317    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.483229 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.483211    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z4qmj"
Sep 16 18:40:06.484289 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.484267    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vjbpc\""
Sep 16 18:40:06.484386 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.484359    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:40:06.485540 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.485516    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:40:06.485636 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.485578    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:40:06.485636 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.485526    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:40:06.485731 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.485528    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:06.486215 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:06.486185    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:06.488235 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488109    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-run-ovn\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.488235 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488146    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-modprobe-d\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.488235 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488185    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-sysctl-conf\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.488465 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488265    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-systemd\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.488465 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488313    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwrsx\" (UniqueName: \"kubernetes.io/projected/346b4c37-63f2-42b6-989f-14bcff205a35-kube-api-access-wwrsx\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.488465 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488374    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be0cf19b-10fb-41c7-be6b-efb9c89b566e-hosts-file\") pod \"node-resolver-tw7bx\" (UID: \"be0cf19b-10fb-41c7-be6b-efb9c89b566e\") " pod="openshift-dns/node-resolver-tw7bx"
Sep 16 18:40:06.488465 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488413    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db04d8f0-efe5-4d33-871b-8a2421f26e17-env-overrides\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.488465 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488443    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db04d8f0-efe5-4d33-871b-8a2421f26e17-ovn-node-metrics-cert\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.488751 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488490    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/346b4c37-63f2-42b6-989f-14bcff205a35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.488751 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488535    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rczbp\""
Sep 16 18:40:06.488751 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488567    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvdx4\" (UniqueName: \"kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4\") pod \"network-check-target-qbtmz\" (UID: \"18bae23f-10bf-41df-a14b-ba453b1ab8f1\") " pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:06.488751 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488605    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5z7l\" (UniqueName: \"kubernetes.io/projected/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-kube-api-access-q5z7l\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.488751 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488640    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-systemd-units\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.488751 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488686    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-socket-dir\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.489031 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488743    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-tmp\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.489031 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488854    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-run-netns\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.489031 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.488895    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/346b4c37-63f2-42b6-989f-14bcff205a35-system-cni-dir\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.489422 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489391    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/346b4c37-63f2-42b6-989f-14bcff205a35-cnibin\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.489536 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489438    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-cni-netd\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.489536 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489466    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jb2f\" (UniqueName: \"kubernetes.io/projected/2b943eaf-4b17-4d8b-9968-a5be25c2c678-kube-api-access-8jb2f\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.489536 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489489    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-sys\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.489536 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489512    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-run-systemd\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.489536 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489535    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-var-lib-kubelet\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.489782 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489559    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cddd1ee3-5425-435b-9607-8d9f2ea8d836-konnectivity-ca\") pod \"konnectivity-agent-9phx7\" (UID: \"cddd1ee3-5425-435b-9607-8d9f2ea8d836\") " pod="kube-system/konnectivity-agent-9phx7"
Sep 16 18:40:06.489782 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489594    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-kubelet\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.489782 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489664    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-run\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.489782 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489688    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55e5f454-d5af-40ab-a53f-b0e12313348b-host-slash\") pod \"iptables-alerter-jzjwf\" (UID: \"55e5f454-d5af-40ab-a53f-b0e12313348b\") " pod="openshift-network-operator/iptables-alerter-jzjwf"
Sep 16 18:40:06.489782 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489711    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-slash\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.489782 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489734    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-node-log\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.489782 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489763    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-log-socket\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.490135 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489786    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db04d8f0-efe5-4d33-871b-8a2421f26e17-ovnkube-config\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.490135 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489810    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44wr5\" (UniqueName: \"kubernetes.io/projected/db04d8f0-efe5-4d33-871b-8a2421f26e17-kube-api-access-44wr5\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.490135 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489849    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-tuned\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.490135 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489882    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/346b4c37-63f2-42b6-989f-14bcff205a35-cni-binary-copy\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.490135 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489911    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55e5f454-d5af-40ab-a53f-b0e12313348b-iptables-alerter-script\") pod \"iptables-alerter-jzjwf\" (UID: \"55e5f454-d5af-40ab-a53f-b0e12313348b\") " pod="openshift-network-operator/iptables-alerter-jzjwf"
Sep 16 18:40:06.490135 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489937    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5vb\" (UniqueName: \"kubernetes.io/projected/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-kube-api-access-7w5vb\") pod \"network-metrics-daemon-zkgpv\" (UID: \"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c\") " pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:06.490135 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489958    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-etc-openvswitch\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.490135 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.489981    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.490135 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490003    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cddd1ee3-5425-435b-9607-8d9f2ea8d836-agent-certs\") pod \"konnectivity-agent-9phx7\" (UID: \"cddd1ee3-5425-435b-9607-8d9f2ea8d836\") " pod="kube-system/konnectivity-agent-9phx7"
Sep 16 18:40:06.490135 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490025    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-sys-fs\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.490135 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490045    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-sysconfig\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.490135 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490066    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-sysctl-d\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.490135 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490107    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0cf19b-10fb-41c7-be6b-efb9c89b566e-tmp-dir\") pod \"node-resolver-tw7bx\" (UID: \"be0cf19b-10fb-41c7-be6b-efb9c89b566e\") " pod="openshift-dns/node-resolver-tw7bx"
Sep 16 18:40:06.490135 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490130    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-var-lib-openvswitch\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490150    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db04d8f0-efe5-4d33-871b-8a2421f26e17-ovnkube-script-lib\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490175    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-registration-dir\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490224    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-etc-selinux\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490256    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-lib-modules\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490280    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-host\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490308    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8crk\" (UniqueName: \"kubernetes.io/projected/be0cf19b-10fb-41c7-be6b-efb9c89b566e-kube-api-access-c8crk\") pod \"node-resolver-tw7bx\" (UID: \"be0cf19b-10fb-41c7-be6b-efb9c89b566e\") " pod="openshift-dns/node-resolver-tw7bx"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490333    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-run-ovn-kubernetes\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490355    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-kubernetes\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490376    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490398    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-device-dir\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490428    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/346b4c37-63f2-42b6-989f-14bcff205a35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490456    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbbl8\" (UniqueName: \"kubernetes.io/projected/55e5f454-d5af-40ab-a53f-b0e12313348b-kube-api-access-qbbl8\") pod \"iptables-alerter-jzjwf\" (UID: \"55e5f454-d5af-40ab-a53f-b0e12313348b\") " pod="openshift-network-operator/iptables-alerter-jzjwf"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490478    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs\") pod \"network-metrics-daemon-zkgpv\" (UID: \"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c\") " pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490527    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-run-openvswitch\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490552    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-cni-bin\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.490781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.490575    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/346b4c37-63f2-42b6-989f-14bcff205a35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.497689 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.497659    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:06.519684 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.519651    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7nrxm"
Sep 16 18:40:06.530698 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.530672    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7nrxm"
Sep 16 18:40:06.579106 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.579058    2552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:40:06.591332 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591299    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cddd1ee3-5425-435b-9607-8d9f2ea8d836-konnectivity-ca\") pod \"konnectivity-agent-9phx7\" (UID: \"cddd1ee3-5425-435b-9607-8d9f2ea8d836\") " pod="kube-system/konnectivity-agent-9phx7"
Sep 16 18:40:06.591332 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591338    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b830482-d9ca-4032-a057-2f87226eed51-cni-binary-copy\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.591602 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591362    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-kubelet\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.591602 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591385    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-run\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.591602 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591435    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55e5f454-d5af-40ab-a53f-b0e12313348b-host-slash\") pod \"iptables-alerter-jzjwf\" (UID: \"55e5f454-d5af-40ab-a53f-b0e12313348b\") " pod="openshift-network-operator/iptables-alerter-jzjwf"
Sep 16 18:40:06.591602 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591502    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-multus-cni-dir\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.591602 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591527    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-slash\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.591602 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591526    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55e5f454-d5af-40ab-a53f-b0e12313348b-host-slash\") pod \"iptables-alerter-jzjwf\" (UID: \"55e5f454-d5af-40ab-a53f-b0e12313348b\") " pod="openshift-network-operator/iptables-alerter-jzjwf"
Sep 16 18:40:06.591602 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591550    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-node-log\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.591602 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591574    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-log-socket\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.591602 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591583    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-kubelet\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.591602 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591597    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db04d8f0-efe5-4d33-871b-8a2421f26e17-ovnkube-config\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591622    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44wr5\" (UniqueName: \"kubernetes.io/projected/db04d8f0-efe5-4d33-871b-8a2421f26e17-kube-api-access-44wr5\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591638    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-run\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591645    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-tuned\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591670    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/346b4c37-63f2-42b6-989f-14bcff205a35-cni-binary-copy\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591695    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55e5f454-d5af-40ab-a53f-b0e12313348b-iptables-alerter-script\") pod \"iptables-alerter-jzjwf\" (UID: \"55e5f454-d5af-40ab-a53f-b0e12313348b\") " pod="openshift-network-operator/iptables-alerter-jzjwf"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591717    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-slash\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591737    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5vb\" (UniqueName: \"kubernetes.io/projected/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-kube-api-access-7w5vb\") pod \"network-metrics-daemon-zkgpv\" (UID: \"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c\") " pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591760    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-etc-openvswitch\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591784    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591821    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cddd1ee3-5425-435b-9607-8d9f2ea8d836-agent-certs\") pod \"konnectivity-agent-9phx7\" (UID: \"cddd1ee3-5425-435b-9607-8d9f2ea8d836\") " pod="kube-system/konnectivity-agent-9phx7"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591846    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-cnibin\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591867    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-multus-socket-dir-parent\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591920    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-sys-fs\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591954    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-sysconfig\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.591977    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-sysctl-d\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592006    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0cf19b-10fb-41c7-be6b-efb9c89b566e-tmp-dir\") pod \"node-resolver-tw7bx\" (UID: \"be0cf19b-10fb-41c7-be6b-efb9c89b566e\") " pod="openshift-dns/node-resolver-tw7bx"
Sep 16 18:40:06.592041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592016    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cddd1ee3-5425-435b-9607-8d9f2ea8d836-konnectivity-ca\") pod \"konnectivity-agent-9phx7\" (UID: \"cddd1ee3-5425-435b-9607-8d9f2ea8d836\") " pod="kube-system/konnectivity-agent-9phx7"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592034    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-var-lib-openvswitch\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592056    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db04d8f0-efe5-4d33-871b-8a2421f26e17-ovnkube-script-lib\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592081    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-registration-dir\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592124    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-etc-selinux\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592146    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-lib-modules\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592184    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-host\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592206    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-system-cni-dir\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592274    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-var-lib-cni-bin\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592310    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b830482-d9ca-4032-a057-2f87226eed51-multus-daemon-config\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592333    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa40654a-058d-4aab-8eeb-b34d81ac854b-host\") pod \"node-ca-z4qmj\" (UID: \"aa40654a-058d-4aab-8eeb-b34d81ac854b\") " pod="openshift-image-registry/node-ca-z4qmj"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592356    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8058be60-2358-4acd-bc5b-6382f4fa2e3e-dbus\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592381    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8crk\" (UniqueName: \"kubernetes.io/projected/be0cf19b-10fb-41c7-be6b-efb9c89b566e-kube-api-access-c8crk\") pod \"node-resolver-tw7bx\" (UID: \"be0cf19b-10fb-41c7-be6b-efb9c89b566e\") " pod="openshift-dns/node-resolver-tw7bx"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592408    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-run-ovn-kubernetes\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592434    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-kubernetes\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592459    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f8pb\" (UniqueName: \"kubernetes.io/projected/3b830482-d9ca-4032-a057-2f87226eed51-kube-api-access-2f8pb\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592486    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-log-socket\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.592724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592515    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.593528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592545    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-var-lib-openvswitch\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.593528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592564    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-device-dir\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.593528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592613    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/346b4c37-63f2-42b6-989f-14bcff205a35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.593528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592640    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbbl8\" (UniqueName: \"kubernetes.io/projected/55e5f454-d5af-40ab-a53f-b0e12313348b-kube-api-access-qbbl8\") pod \"iptables-alerter-jzjwf\" (UID: \"55e5f454-d5af-40ab-a53f-b0e12313348b\") " pod="openshift-network-operator/iptables-alerter-jzjwf"
Sep 16 18:40:06.593528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592665    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-run-k8s-cni-cncf-io\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.593528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592668    2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:40:06.593528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592693    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db04d8f0-efe5-4d33-871b-8a2421f26e17-ovnkube-config\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.593528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.592690    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-etc-kubernetes\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.593528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593227    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/346b4c37-63f2-42b6-989f-14bcff205a35-cni-binary-copy\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.593528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593269    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55e5f454-d5af-40ab-a53f-b0e12313348b-iptables-alerter-script\") pod \"iptables-alerter-jzjwf\" (UID: \"55e5f454-d5af-40ab-a53f-b0e12313348b\") " pod="openshift-network-operator/iptables-alerter-jzjwf"
Sep 16 18:40:06.593528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593269    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db04d8f0-efe5-4d33-871b-8a2421f26e17-ovnkube-script-lib\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.593528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593303    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-etc-openvswitch\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.593528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593345    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.593528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593409    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-registration-dir\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.593528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593459    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-etc-selinux\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593583    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-run-ovn-kubernetes\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593601    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-sysconfig\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593626    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-lib-modules\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593672    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-sys-fs\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593673    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-host\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593688    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-kubernetes\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593737    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593740    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-device-dir\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593805    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-node-log\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593892    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-sysctl-d\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593903    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs\") pod \"network-metrics-daemon-zkgpv\" (UID: \"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c\") " pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593944    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-run-openvswitch\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593972    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-cni-bin\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.593997    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/346b4c37-63f2-42b6-989f-14bcff205a35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594024    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0cf19b-10fb-41c7-be6b-efb9c89b566e-tmp-dir\") pod \"node-resolver-tw7bx\" (UID: \"be0cf19b-10fb-41c7-be6b-efb9c89b566e\") " pod="openshift-dns/node-resolver-tw7bx"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594026    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-var-lib-cni-multus\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.594238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594114    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-run-multus-certs\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594148    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-run-ovn\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:06.594161    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594174    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-modprobe-d\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594199    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-sysctl-conf\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594224    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-systemd\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594249    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwrsx\" (UniqueName: \"kubernetes.io/projected/346b4c37-63f2-42b6-989f-14bcff205a35-kube-api-access-wwrsx\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:06.594267    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs podName:0f5f1c97-1813-4d89-8af7-ce7d0eebc10c nodeName:}" failed. No retries permitted until 2026-09-16 18:40:07.094245319 +0000 UTC m=+2.988432571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs") pod "network-metrics-daemon-zkgpv" (UID: "0f5f1c97-1813-4d89-8af7-ce7d0eebc10c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594298    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be0cf19b-10fb-41c7-be6b-efb9c89b566e-hosts-file\") pod \"node-resolver-tw7bx\" (UID: \"be0cf19b-10fb-41c7-be6b-efb9c89b566e\") " pod="openshift-dns/node-resolver-tw7bx"
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594326    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db04d8f0-efe5-4d33-871b-8a2421f26e17-env-overrides\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594352    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db04d8f0-efe5-4d33-871b-8a2421f26e17-ovn-node-metrics-cert\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594377    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/346b4c37-63f2-42b6-989f-14bcff205a35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594407    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdx4\" (UniqueName: \"kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4\") pod \"network-check-target-qbtmz\" (UID: \"18bae23f-10bf-41df-a14b-ba453b1ab8f1\") " pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594422    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/346b4c37-63f2-42b6-989f-14bcff205a35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594433    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5z7l\" (UniqueName: \"kubernetes.io/projected/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-kube-api-access-q5z7l\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594466    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-run-netns\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594492    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-var-lib-kubelet\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.594997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594498    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-run-ovn\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594514    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-hostroot\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594544    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-systemd-units\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594555    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-systemd\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594573    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-socket-dir\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594596    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-tmp\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594605    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-modprobe-d\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594625    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-multus-conf-dir\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594661    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-sysctl-conf\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594669    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-systemd-units\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594707    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa40654a-058d-4aab-8eeb-b34d81ac854b-serviceca\") pod \"node-ca-z4qmj\" (UID: \"aa40654a-058d-4aab-8eeb-b34d81ac854b\") " pod="openshift-image-registry/node-ca-z4qmj"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594732    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzxbv\" (UniqueName: \"kubernetes.io/projected/aa40654a-058d-4aab-8eeb-b34d81ac854b-kube-api-access-bzxbv\") pod \"node-ca-z4qmj\" (UID: \"aa40654a-058d-4aab-8eeb-b34d81ac854b\") " pod="openshift-image-registry/node-ca-z4qmj"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594764    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-run-netns\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594799    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/346b4c37-63f2-42b6-989f-14bcff205a35-system-cni-dir\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594825    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/346b4c37-63f2-42b6-989f-14bcff205a35-cnibin\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594848    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-cni-netd\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594872    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jb2f\" (UniqueName: \"kubernetes.io/projected/2b943eaf-4b17-4d8b-9968-a5be25c2c678-kube-api-access-8jb2f\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.595827 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594897    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-sys\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594921    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8058be60-2358-4acd-bc5b-6382f4fa2e3e-kubelet-config\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594940    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/346b4c37-63f2-42b6-989f-14bcff205a35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.594947    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.595038    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-run-systemd\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.595070    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-var-lib-kubelet\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.595183    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-var-lib-kubelet\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.595188    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/346b4c37-63f2-42b6-989f-14bcff205a35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.595205    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/346b4c37-63f2-42b6-989f-14bcff205a35-system-cni-dir\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.595219    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/346b4c37-63f2-42b6-989f-14bcff205a35-cnibin\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.595247    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-cni-bin\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.595288    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-run-netns\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.595299    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b943eaf-4b17-4d8b-9968-a5be25c2c678-socket-dir\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.595338    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-run-openvswitch\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.595424    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-sys\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.595486    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be0cf19b-10fb-41c7-be6b-efb9c89b566e-hosts-file\") pod \"node-resolver-tw7bx\" (UID: \"be0cf19b-10fb-41c7-be6b-efb9c89b566e\") " pod="openshift-dns/node-resolver-tw7bx"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.595482    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-host-cni-netd\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.596655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.595542    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db04d8f0-efe5-4d33-871b-8a2421f26e17-run-systemd\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.597446 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.595674    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db04d8f0-efe5-4d33-871b-8a2421f26e17-env-overrides\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.597446 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.597244    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-etc-tuned\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.597712 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.597522    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db04d8f0-efe5-4d33-871b-8a2421f26e17-ovn-node-metrics-cert\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.597712 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.597609    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cddd1ee3-5425-435b-9607-8d9f2ea8d836-agent-certs\") pod \"konnectivity-agent-9phx7\" (UID: \"cddd1ee3-5425-435b-9607-8d9f2ea8d836\") " pod="kube-system/konnectivity-agent-9phx7"
Sep 16 18:40:06.598167 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.598136    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-tmp\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.601912 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:06.601883    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:06.601912 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:06.601911    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:06.602124 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:06.601927    2552 projected.go:196] Error preparing data for projected volume kube-api-access-hvdx4 for pod openshift-network-diagnostics/network-check-target-qbtmz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:06.602124 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:06.601994    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4 podName:18bae23f-10bf-41df-a14b-ba453b1ab8f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:07.101972921 +0000 UTC m=+2.996160169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hvdx4" (UniqueName: "kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4") pod "network-check-target-qbtmz" (UID: "18bae23f-10bf-41df-a14b-ba453b1ab8f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:06.606959 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.606921    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5z7l\" (UniqueName: \"kubernetes.io/projected/19d74f52-34af-4eb4-8a00-7c5c9e66b2c7-kube-api-access-q5z7l\") pod \"tuned-89jvz\" (UID: \"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7\") " pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.607236 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.607209    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbbl8\" (UniqueName: \"kubernetes.io/projected/55e5f454-d5af-40ab-a53f-b0e12313348b-kube-api-access-qbbl8\") pod \"iptables-alerter-jzjwf\" (UID: \"55e5f454-d5af-40ab-a53f-b0e12313348b\") " pod="openshift-network-operator/iptables-alerter-jzjwf"
Sep 16 18:40:06.607383 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.607360    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5vb\" (UniqueName: \"kubernetes.io/projected/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-kube-api-access-7w5vb\") pod \"network-metrics-daemon-zkgpv\" (UID: \"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c\") " pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:06.607460 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.607400    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwrsx\" (UniqueName: \"kubernetes.io/projected/346b4c37-63f2-42b6-989f-14bcff205a35-kube-api-access-wwrsx\") pod \"multus-additional-cni-plugins-rnccg\" (UID: \"346b4c37-63f2-42b6-989f-14bcff205a35\") " pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.607711 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.607691    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44wr5\" (UniqueName: \"kubernetes.io/projected/db04d8f0-efe5-4d33-871b-8a2421f26e17-kube-api-access-44wr5\") pod \"ovnkube-node-cs8zv\" (UID: \"db04d8f0-efe5-4d33-871b-8a2421f26e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.608191 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.608169    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8crk\" (UniqueName: \"kubernetes.io/projected/be0cf19b-10fb-41c7-be6b-efb9c89b566e-kube-api-access-c8crk\") pod \"node-resolver-tw7bx\" (UID: \"be0cf19b-10fb-41c7-be6b-efb9c89b566e\") " pod="openshift-dns/node-resolver-tw7bx"
Sep 16 18:40:06.609507 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.609483    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jb2f\" (UniqueName: \"kubernetes.io/projected/2b943eaf-4b17-4d8b-9968-a5be25c2c678-kube-api-access-8jb2f\") pod \"aws-ebs-csi-driver-node-p2bbd\" (UID: \"2b943eaf-4b17-4d8b-9968-a5be25c2c678\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.695811 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.695771    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8058be60-2358-4acd-bc5b-6382f4fa2e3e-kubelet-config\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:06.696021 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.695821    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:06.696021 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.695846    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b830482-d9ca-4032-a057-2f87226eed51-cni-binary-copy\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696021 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.695873    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-multus-cni-dir\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696021 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.695902    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-cnibin\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696021 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.695923    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-multus-socket-dir-parent\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696021 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.695916    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8058be60-2358-4acd-bc5b-6382f4fa2e3e-kubelet-config\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:06.696021 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:06.695941    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:06.696021 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.695966    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-cnibin\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696021 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696004    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-multus-cni-dir\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696019    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-multus-socket-dir-parent\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:06.696020    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret podName:8058be60-2358-4acd-bc5b-6382f4fa2e3e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:07.196000305 +0000 UTC m=+3.090187558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret") pod "global-pull-secret-syncer-g5lz9" (UID: "8058be60-2358-4acd-bc5b-6382f4fa2e3e") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696072    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-system-cni-dir\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696114    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-var-lib-cni-bin\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696137    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b830482-d9ca-4032-a057-2f87226eed51-multus-daemon-config\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696168    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa40654a-058d-4aab-8eeb-b34d81ac854b-host\") pod \"node-ca-z4qmj\" (UID: \"aa40654a-058d-4aab-8eeb-b34d81ac854b\") " pod="openshift-image-registry/node-ca-z4qmj"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696175    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-system-cni-dir\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696192    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8058be60-2358-4acd-bc5b-6382f4fa2e3e-dbus\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696229    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-var-lib-cni-bin\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696230    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f8pb\" (UniqueName: \"kubernetes.io/projected/3b830482-d9ca-4032-a057-2f87226eed51-kube-api-access-2f8pb\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696274    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-run-k8s-cni-cncf-io\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696298    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-etc-kubernetes\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696315    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8058be60-2358-4acd-bc5b-6382f4fa2e3e-dbus\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696333    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-var-lib-cni-multus\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696351    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-run-k8s-cni-cncf-io\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696355    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-run-multus-certs\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.696508 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696235    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa40654a-058d-4aab-8eeb-b34d81ac854b-host\") pod \"node-ca-z4qmj\" (UID: \"aa40654a-058d-4aab-8eeb-b34d81ac854b\") " pod="openshift-image-registry/node-ca-z4qmj"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696391    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-run-multus-certs\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696397    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-run-netns\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696417    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-run-netns\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696422    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-var-lib-kubelet\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696441    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-etc-kubernetes\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696445    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-hostroot\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696464    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-var-lib-cni-multus\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696475    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-multus-conf-dir\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696480    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-host-var-lib-kubelet\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696487    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-hostroot\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696497    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa40654a-058d-4aab-8eeb-b34d81ac854b-serviceca\") pod \"node-ca-z4qmj\" (UID: \"aa40654a-058d-4aab-8eeb-b34d81ac854b\") " pod="openshift-image-registry/node-ca-z4qmj"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696521    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzxbv\" (UniqueName: \"kubernetes.io/projected/aa40654a-058d-4aab-8eeb-b34d81ac854b-kube-api-access-bzxbv\") pod \"node-ca-z4qmj\" (UID: \"aa40654a-058d-4aab-8eeb-b34d81ac854b\") " pod="openshift-image-registry/node-ca-z4qmj"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696523    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b830482-d9ca-4032-a057-2f87226eed51-multus-conf-dir\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696594    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b830482-d9ca-4032-a057-2f87226eed51-cni-binary-copy\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696714    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b830482-d9ca-4032-a057-2f87226eed51-multus-daemon-config\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.697189 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.696891    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa40654a-058d-4aab-8eeb-b34d81ac854b-serviceca\") pod \"node-ca-z4qmj\" (UID: \"aa40654a-058d-4aab-8eeb-b34d81ac854b\") " pod="openshift-image-registry/node-ca-z4qmj"
Sep 16 18:40:06.707469 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.707398    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzxbv\" (UniqueName: \"kubernetes.io/projected/aa40654a-058d-4aab-8eeb-b34d81ac854b-kube-api-access-bzxbv\") pod \"node-ca-z4qmj\" (UID: \"aa40654a-058d-4aab-8eeb-b34d81ac854b\") " pod="openshift-image-registry/node-ca-z4qmj"
Sep 16 18:40:06.707469 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.707446    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f8pb\" (UniqueName: \"kubernetes.io/projected/3b830482-d9ca-4032-a057-2f87226eed51-kube-api-access-2f8pb\") pod \"multus-zvqnt\" (UID: \"3b830482-d9ca-4032-a057-2f87226eed51\") " pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.772492 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.772446    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:06.787925 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.787880    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd"
Sep 16 18:40:06.796372 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.796336    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tw7bx"
Sep 16 18:40:06.805371 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.805343    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-89jvz"
Sep 16 18:40:06.813271 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.813226    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rnccg"
Sep 16 18:40:06.822564 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.822529    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jzjwf"
Sep 16 18:40:06.830666 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.830630    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:06.837737 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.837698    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9phx7"
Sep 16 18:40:06.845164 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.845126    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zvqnt"
Sep 16 18:40:06.851167 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:06.851130    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z4qmj"
Sep 16 18:40:07.100682 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:07.100594    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs\") pod \"network-metrics-daemon-zkgpv\" (UID: \"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c\") " pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:07.100846 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:07.100792    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:07.100905 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:07.100883    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs podName:0f5f1c97-1813-4d89-8af7-ce7d0eebc10c nodeName:}" failed. No retries permitted until 2026-09-16 18:40:08.100860136 +0000 UTC m=+3.995047390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs") pod "network-metrics-daemon-zkgpv" (UID: "0f5f1c97-1813-4d89-8af7-ce7d0eebc10c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:07.202132 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:07.201790    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdx4\" (UniqueName: \"kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4\") pod \"network-check-target-qbtmz\" (UID: \"18bae23f-10bf-41df-a14b-ba453b1ab8f1\") " pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:07.202132 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:07.201865    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:07.202132 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:07.201971    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:07.202132 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:07.201996    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:07.202132 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:07.201997    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:07.202132 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:07.202010    2552 projected.go:196] Error preparing data for projected volume kube-api-access-hvdx4 for pod openshift-network-diagnostics/network-check-target-qbtmz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:07.202132 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:07.202074    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret podName:8058be60-2358-4acd-bc5b-6382f4fa2e3e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:08.202051512 +0000 UTC m=+4.096238783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret") pod "global-pull-secret-syncer-g5lz9" (UID: "8058be60-2358-4acd-bc5b-6382f4fa2e3e") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:07.202132 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:07.202098    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4 podName:18bae23f-10bf-41df-a14b-ba453b1ab8f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:08.202087416 +0000 UTC m=+4.096274685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvdx4" (UniqueName: "kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4") pod "network-check-target-qbtmz" (UID: "18bae23f-10bf-41df-a14b-ba453b1ab8f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:07.533321 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:07.533243    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:07 +0000 UTC" deadline="2028-03-17 12:47:22.414454204 +0000 UTC"
Sep 16 18:40:07.533321 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:07.533319    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13146h7m14.881142361s"
Sep 16 18:40:07.643323 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:07.642832    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa40654a_058d_4aab_8eeb_b34d81ac854b.slice/crio-16915e33778483b29b7007e834a86c0f10b3a71686997cfc4edc75e02483f11c WatchSource:0}: Error finding container 16915e33778483b29b7007e834a86c0f10b3a71686997cfc4edc75e02483f11c: Status 404 returned error can't find the container with id 16915e33778483b29b7007e834a86c0f10b3a71686997cfc4edc75e02483f11c
Sep 16 18:40:07.644886 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:07.644853    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0cf19b_10fb_41c7_be6b_efb9c89b566e.slice/crio-c624428a750668e22912f58de7128a8f4a82fc693adb7cd928d2dcfa4d317399 WatchSource:0}: Error finding container c624428a750668e22912f58de7128a8f4a82fc693adb7cd928d2dcfa4d317399: Status 404 returned error can't find the container with id c624428a750668e22912f58de7128a8f4a82fc693adb7cd928d2dcfa4d317399
Sep 16 18:40:07.645836 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:07.645811    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb04d8f0_efe5_4d33_871b_8a2421f26e17.slice/crio-2cab88a39796f0ad01d82cfbedfaadea734d2c2a92f0c07986c303e8fcf176b7 WatchSource:0}: Error finding container 2cab88a39796f0ad01d82cfbedfaadea734d2c2a92f0c07986c303e8fcf176b7: Status 404 returned error can't find the container with id 2cab88a39796f0ad01d82cfbedfaadea734d2c2a92f0c07986c303e8fcf176b7
Sep 16 18:40:07.649019 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:07.648713    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddd1ee3_5425_435b_9607_8d9f2ea8d836.slice/crio-fc9b4fb7065ee0e5d42051f5bba4229c4d29939fc8eece060deef2a5cab5afa5 WatchSource:0}: Error finding container fc9b4fb7065ee0e5d42051f5bba4229c4d29939fc8eece060deef2a5cab5afa5: Status 404 returned error can't find the container with id fc9b4fb7065ee0e5d42051f5bba4229c4d29939fc8eece060deef2a5cab5afa5
Sep 16 18:40:08.111359 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.110418    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs\") pod \"network-metrics-daemon-zkgpv\" (UID: \"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c\") " pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:08.111359 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:08.110698    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:08.111359 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:08.110813    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs podName:0f5f1c97-1813-4d89-8af7-ce7d0eebc10c nodeName:}" failed. No retries permitted until 2026-09-16 18:40:10.110753626 +0000 UTC m=+6.004940893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs") pod "network-metrics-daemon-zkgpv" (UID: "0f5f1c97-1813-4d89-8af7-ce7d0eebc10c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:08.212422 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.211597    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdx4\" (UniqueName: \"kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4\") pod \"network-check-target-qbtmz\" (UID: \"18bae23f-10bf-41df-a14b-ba453b1ab8f1\") " pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:08.212422 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.211663    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:08.212422 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:08.211799    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:08.212422 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:08.211870    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret podName:8058be60-2358-4acd-bc5b-6382f4fa2e3e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:10.211849835 +0000 UTC m=+6.106037099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret") pod "global-pull-secret-syncer-g5lz9" (UID: "8058be60-2358-4acd-bc5b-6382f4fa2e3e") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:08.212422 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:08.211965    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:08.212422 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:08.211979    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:08.212422 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:08.211994    2552 projected.go:196] Error preparing data for projected volume kube-api-access-hvdx4 for pod openshift-network-diagnostics/network-check-target-qbtmz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:08.212422 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:08.212032    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4 podName:18bae23f-10bf-41df-a14b-ba453b1ab8f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:10.212018834 +0000 UTC m=+6.106206094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvdx4" (UniqueName: "kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4") pod "network-check-target-qbtmz" (UID: "18bae23f-10bf-41df-a14b-ba453b1ab8f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:08.534236 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.534155    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:07 +0000 UTC" deadline="2028-06-19 20:15:34.85492051 +0000 UTC"
Sep 16 18:40:08.534236 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.534234    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15409h35m26.320691728s"
Sep 16 18:40:08.619736 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.619699    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:08.619927 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.619748    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:08.619927 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.619699    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:08.619927 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:08.619870    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:08.620127 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:08.619953    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:08.620127 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:08.620048    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:08.645752 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.645650    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnccg" event={"ID":"346b4c37-63f2-42b6-989f-14bcff205a35","Type":"ContainerStarted","Data":"fc8941997730c2ab8987bfeba2eed1b14fc92505abc8d2c5b821fb4663d4fe1a"}
Sep 16 18:40:08.660222 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.660147    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9phx7" event={"ID":"cddd1ee3-5425-435b-9607-8d9f2ea8d836","Type":"ContainerStarted","Data":"fc9b4fb7065ee0e5d42051f5bba4229c4d29939fc8eece060deef2a5cab5afa5"}
Sep 16 18:40:08.665552 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.665504    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tw7bx" event={"ID":"be0cf19b-10fb-41c7-be6b-efb9c89b566e","Type":"ContainerStarted","Data":"c624428a750668e22912f58de7128a8f4a82fc693adb7cd928d2dcfa4d317399"}
Sep 16 18:40:08.671185 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.671117    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv" event={"ID":"db04d8f0-efe5-4d33-871b-8a2421f26e17","Type":"ContainerStarted","Data":"2cab88a39796f0ad01d82cfbedfaadea734d2c2a92f0c07986c303e8fcf176b7"}
Sep 16 18:40:08.684651 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.684604    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-72.ec2.internal" event={"ID":"8b97e76fa4a597a2b9f96eb244b741d8","Type":"ContainerStarted","Data":"01b2c99fe442bd9e62047970eb01158fb62bced065479e8b907597f0aceb2651"}
Sep 16 18:40:08.702392 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.702108    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zvqnt" event={"ID":"3b830482-d9ca-4032-a057-2f87226eed51","Type":"ContainerStarted","Data":"b09c3f22914f1c250c6a63f8f358ad7c238d1d31aa999a24eae5aff29ae59a86"}
Sep 16 18:40:08.709052 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.708858    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jzjwf" event={"ID":"55e5f454-d5af-40ab-a53f-b0e12313348b","Type":"ContainerStarted","Data":"493d5e1736de99690ff67ea2221c9f3b924fac9d0a63abfbae5f31d90aba089c"}
Sep 16 18:40:08.716881 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.716754    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd" event={"ID":"2b943eaf-4b17-4d8b-9968-a5be25c2c678","Type":"ContainerStarted","Data":"b33232560c6d46c18a80dff5fc7bf079b207ce185e76b95472f7c4b232620f43"}
Sep 16 18:40:08.722920 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.722804    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z4qmj" event={"ID":"aa40654a-058d-4aab-8eeb-b34d81ac854b","Type":"ContainerStarted","Data":"16915e33778483b29b7007e834a86c0f10b3a71686997cfc4edc75e02483f11c"}
Sep 16 18:40:08.733727 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.733680    2552 generic.go:356] "Generic (PLEG): container finished" podID="5b809a87ee2a1bceb0e61eaa5e49cd69" containerID="2ce770bbb7d0657e4242a1c87deef2b45572c7d5ff21dcff3c58787c55064bce" exitCode=0
Sep 16 18:40:08.733918 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.733806    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal" event={"ID":"5b809a87ee2a1bceb0e61eaa5e49cd69","Type":"ContainerDied","Data":"2ce770bbb7d0657e4242a1c87deef2b45572c7d5ff21dcff3c58787c55064bce"}
Sep 16 18:40:08.754410 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.754280    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-72.ec2.internal" podStartSLOduration=2.754253115 podStartE2EDuration="2.754253115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:08.707923876 +0000 UTC m=+4.602111150" watchObservedRunningTime="2026-09-16 18:40:08.754253115 +0000 UTC m=+4.648440391"
Sep 16 18:40:08.760409 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:08.760365    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-89jvz" event={"ID":"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7","Type":"ContainerStarted","Data":"8630345dcd76479fab28a075b66b25ceb4f26f41689774ba60a432e01d8a4c07"}
Sep 16 18:40:09.782086 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:09.781982    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal" event={"ID":"5b809a87ee2a1bceb0e61eaa5e49cd69","Type":"ContainerStarted","Data":"6243b93bf7054135312be4032c10e592697dff17114608428146010b8c921c85"}
Sep 16 18:40:09.807634 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:09.806754    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-72.ec2.internal" podStartSLOduration=3.8067301430000002 podStartE2EDuration="3.806730143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:09.806483671 +0000 UTC m=+5.700670944" watchObservedRunningTime="2026-09-16 18:40:09.806730143 +0000 UTC m=+5.700917418"
Sep 16 18:40:10.133019 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:10.132975    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs\") pod \"network-metrics-daemon-zkgpv\" (UID: \"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c\") " pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:10.133262 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:10.133159    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:10.133262 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:10.133230    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs podName:0f5f1c97-1813-4d89-8af7-ce7d0eebc10c nodeName:}" failed. No retries permitted until 2026-09-16 18:40:14.133210854 +0000 UTC m=+10.027398118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs") pod "network-metrics-daemon-zkgpv" (UID: "0f5f1c97-1813-4d89-8af7-ce7d0eebc10c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:10.233899 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:10.233814    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdx4\" (UniqueName: \"kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4\") pod \"network-check-target-qbtmz\" (UID: \"18bae23f-10bf-41df-a14b-ba453b1ab8f1\") " pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:10.233899 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:10.233883    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:10.234195 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:10.234029    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:10.234195 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:10.234097    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret podName:8058be60-2358-4acd-bc5b-6382f4fa2e3e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:14.234077913 +0000 UTC m=+10.128265184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret") pod "global-pull-secret-syncer-g5lz9" (UID: "8058be60-2358-4acd-bc5b-6382f4fa2e3e") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:10.234617 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:10.234594    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:10.234720 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:10.234626    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:10.234720 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:10.234641    2552 projected.go:196] Error preparing data for projected volume kube-api-access-hvdx4 for pod openshift-network-diagnostics/network-check-target-qbtmz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:10.234720 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:10.234688    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4 podName:18bae23f-10bf-41df-a14b-ba453b1ab8f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:14.234672126 +0000 UTC m=+10.128859378 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvdx4" (UniqueName: "kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4") pod "network-check-target-qbtmz" (UID: "18bae23f-10bf-41df-a14b-ba453b1ab8f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:10.622962 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:10.622067    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:10.622962 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:10.622243    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:10.622962 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:10.622636    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:10.622962 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:10.622730    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:10.622962 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:10.622785    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:10.622962 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:10.622859    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:12.620856 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:12.620301    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:12.620856 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:12.620461    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:12.620856 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:12.620508    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:12.620856 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:12.620627    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:12.620856 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:12.620516    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:12.620856 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:12.620795    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:14.170505 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:14.170371    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs\") pod \"network-metrics-daemon-zkgpv\" (UID: \"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c\") " pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:14.171021 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:14.170513    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:14.171021 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:14.170582    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs podName:0f5f1c97-1813-4d89-8af7-ce7d0eebc10c nodeName:}" failed. No retries permitted until 2026-09-16 18:40:22.170563462 +0000 UTC m=+18.064750719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs") pod "network-metrics-daemon-zkgpv" (UID: "0f5f1c97-1813-4d89-8af7-ce7d0eebc10c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:14.271628 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:14.271582    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdx4\" (UniqueName: \"kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4\") pod \"network-check-target-qbtmz\" (UID: \"18bae23f-10bf-41df-a14b-ba453b1ab8f1\") " pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:14.271828 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:14.271653    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:14.271902 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:14.271831    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:14.271963 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:14.271905    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret podName:8058be60-2358-4acd-bc5b-6382f4fa2e3e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:22.271883688 +0000 UTC m=+18.166071025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret") pod "global-pull-secret-syncer-g5lz9" (UID: "8058be60-2358-4acd-bc5b-6382f4fa2e3e") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:14.272464 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:14.272395    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:14.272464 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:14.272419    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:14.272464 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:14.272433    2552 projected.go:196] Error preparing data for projected volume kube-api-access-hvdx4 for pod openshift-network-diagnostics/network-check-target-qbtmz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:14.272664 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:14.272486    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4 podName:18bae23f-10bf-41df-a14b-ba453b1ab8f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:22.272470985 +0000 UTC m=+18.166658233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvdx4" (UniqueName: "kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4") pod "network-check-target-qbtmz" (UID: "18bae23f-10bf-41df-a14b-ba453b1ab8f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:14.619643 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:14.619533    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:14.619814 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:14.619694    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:14.620041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:14.620013    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:14.621387 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:14.621185    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:14.621387 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:14.621271    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:14.621387 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:14.621355    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:16.620242 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:16.619932    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:16.620709 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:16.620384    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:16.620709 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:16.619974    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:16.620709 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:16.620042    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:16.620709 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:16.620521    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:16.620709 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:16.620613    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:18.619884 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:18.619843    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:18.620387 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:18.619843    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:18.620387 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:18.619988    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:18.620387 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:18.620006    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:18.620387 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:18.620133    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:18.620387 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:18.620241    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:20.622696 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:20.622664    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:20.623118 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:20.622770    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:20.623118 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:20.622798    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:20.623118 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:20.622785    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:20.623118 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:20.622896    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:20.623118 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:20.622962    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:22.236972 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:22.236928    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs\") pod \"network-metrics-daemon-zkgpv\" (UID: \"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c\") " pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:22.237527 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:22.237144    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:22.237527 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:22.237233    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs podName:0f5f1c97-1813-4d89-8af7-ce7d0eebc10c nodeName:}" failed. No retries permitted until 2026-09-16 18:40:38.237210293 +0000 UTC m=+34.131397548 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs") pod "network-metrics-daemon-zkgpv" (UID: "0f5f1c97-1813-4d89-8af7-ce7d0eebc10c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:22.337345 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:22.337307    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:22.337552 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:22.337408    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdx4\" (UniqueName: \"kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4\") pod \"network-check-target-qbtmz\" (UID: \"18bae23f-10bf-41df-a14b-ba453b1ab8f1\") " pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:22.337552 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:22.337487    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:22.337552 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:22.337546    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:22.337713 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:22.337566    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:22.337713 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:22.337576    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret podName:8058be60-2358-4acd-bc5b-6382f4fa2e3e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:38.337558521 +0000 UTC m=+34.231745781 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret") pod "global-pull-secret-syncer-g5lz9" (UID: "8058be60-2358-4acd-bc5b-6382f4fa2e3e") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:22.337713 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:22.337580    2552 projected.go:196] Error preparing data for projected volume kube-api-access-hvdx4 for pod openshift-network-diagnostics/network-check-target-qbtmz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:22.337713 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:22.337624    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4 podName:18bae23f-10bf-41df-a14b-ba453b1ab8f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:38.337614018 +0000 UTC m=+34.231801266 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvdx4" (UniqueName: "kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4") pod "network-check-target-qbtmz" (UID: "18bae23f-10bf-41df-a14b-ba453b1ab8f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:22.619427 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:22.619337    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:22.619427 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:22.619337    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:22.619427 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:22.619337    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:22.619707 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:22.619478    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:22.619707 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:22.619531    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:22.619707 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:22.619582    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:24.620208 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:24.620157    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:24.620730 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:24.620257    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:24.620730 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:24.620265    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:24.620730 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:24.620394    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:24.620730 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:24.620429    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:24.620730 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:24.620505    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:26.619702 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:26.619617    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:26.619702 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:26.619638    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:26.620197 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:26.619777    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:26.620197 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:26.619818    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:26.620197 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:26.619892    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:26.620362 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:26.620321    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:26.822857 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:26.822662    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zvqnt" event={"ID":"3b830482-d9ca-4032-a057-2f87226eed51","Type":"ContainerStarted","Data":"03175305490d1152d105a992668d642057015a1d7178e8e0c8d301799223d9fc"}
Sep 16 18:40:26.828102 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:26.827862    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z4qmj" event={"ID":"aa40654a-058d-4aab-8eeb-b34d81ac854b","Type":"ContainerStarted","Data":"4ab30913d64ae8f29df2a2250dd21182987132c9cd7ec20dc4eade72e7013edc"}
Sep 16 18:40:26.832981 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:26.832895    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-89jvz" event={"ID":"19d74f52-34af-4eb4-8a00-7c5c9e66b2c7","Type":"ContainerStarted","Data":"676ba2b9a1e509de2227659a561899002fcfc0c4ae0c81f93221ab69f1050372"}
Sep 16 18:40:26.839541 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:26.839502    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tw7bx" event={"ID":"be0cf19b-10fb-41c7-be6b-efb9c89b566e","Type":"ContainerStarted","Data":"709fdbe05b70e870e1d6b9fefda0a295970a56e47fe3e3e83f068b853f0bbb25"}
Sep 16 18:40:26.844224 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:26.843984    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv" event={"ID":"db04d8f0-efe5-4d33-871b-8a2421f26e17","Type":"ContainerStarted","Data":"da0dcaf225b06a808d881aae5dd60fec448dfdf3087af42932d82c1ae74793f2"}
Sep 16 18:40:26.853611 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:26.853189    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zvqnt" podStartSLOduration=2.841634032 podStartE2EDuration="21.853167158s" totalImagesPullingTime="19.011533126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:26.852569037 +0000 UTC m=+22.746756308" watchObservedRunningTime="2026-09-16 18:40:26.853167158 +0000 UTC m=+22.747354427"
Sep 16 18:40:26.875294 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:26.875244    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-89jvz" podStartSLOduration=2.926251674 podStartE2EDuration="21.875227291s" totalImagesPullingTime="18.948975617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:26.874553294 +0000 UTC m=+22.768740565" watchObservedRunningTime="2026-09-16 18:40:26.875227291 +0000 UTC m=+22.769414559"
Sep 16 18:40:26.888868 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:26.888811    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z4qmj" podStartSLOduration=1.944225952 podStartE2EDuration="20.88879182s" totalImagesPullingTime="18.944565868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:26.8877471 +0000 UTC m=+22.781934370" watchObservedRunningTime="2026-09-16 18:40:26.88879182 +0000 UTC m=+22.782979089"
Sep 16 18:40:26.934469 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:26.934419    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tw7bx" podStartSLOduration=2.998898031 podStartE2EDuration="21.934401071s" totalImagesPullingTime="18.93550304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:26.915613403 +0000 UTC m=+22.809800673" watchObservedRunningTime="2026-09-16 18:40:26.934401071 +0000 UTC m=+22.828588340"
Sep 16 18:40:27.849594 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:27.849552    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv" event={"ID":"db04d8f0-efe5-4d33-871b-8a2421f26e17","Type":"ContainerStarted","Data":"2d880fc1fcf7a355d7f6dfb43b144b620532668f653684f10d8f4aaeecdac3e9"}
Sep 16 18:40:27.850491 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:27.849670    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv" event={"ID":"db04d8f0-efe5-4d33-871b-8a2421f26e17","Type":"ContainerStarted","Data":"44e2a7b80c46d3247ce9ac9b96da333adc5130d3b1d303ba4f0ea9ef6f8de06b"}
Sep 16 18:40:27.850491 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:27.849721    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv" event={"ID":"db04d8f0-efe5-4d33-871b-8a2421f26e17","Type":"ContainerStarted","Data":"f0331b1596d70de0c1b7ac1a96b1e028c829262c42cb1cafd9819a7e340b42b4"}
Sep 16 18:40:27.850491 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:27.849769    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv" event={"ID":"db04d8f0-efe5-4d33-871b-8a2421f26e17","Type":"ContainerStarted","Data":"4cdab68782c2325f9b5f39d9a5a2f2444fbb2e373d8b787906457c384abcc504"}
Sep 16 18:40:27.850491 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:27.849818    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv" event={"ID":"db04d8f0-efe5-4d33-871b-8a2421f26e17","Type":"ContainerStarted","Data":"90933f7d4a65bd4d385c730a36e3794d1a807ab7efc4e28a372452e15b292faa"}
Sep 16 18:40:27.851121 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:27.851082    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd" event={"ID":"2b943eaf-4b17-4d8b-9968-a5be25c2c678","Type":"ContainerStarted","Data":"f13fcaea1f45368472244c4f7a463422ae0c09e0bb83c9fce2f3e81702d8070c"}
Sep 16 18:40:27.852451 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:27.852425    2552 generic.go:356] "Generic (PLEG): container finished" podID="346b4c37-63f2-42b6-989f-14bcff205a35" containerID="116aae8bf8e7833823185a14c961cad3af19906b4c511f9d567c0046da50e934" exitCode=0
Sep 16 18:40:27.852569 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:27.852493    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnccg" event={"ID":"346b4c37-63f2-42b6-989f-14bcff205a35","Type":"ContainerDied","Data":"116aae8bf8e7833823185a14c961cad3af19906b4c511f9d567c0046da50e934"}
Sep 16 18:40:27.854195 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:27.854165    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9phx7" event={"ID":"cddd1ee3-5425-435b-9607-8d9f2ea8d836","Type":"ContainerStarted","Data":"a16eab77fa66db8174e1912eb3c44563f602183eeb8009e724d36091fc1ce2ab"}
Sep 16 18:40:27.894446 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:27.894390    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9phx7" podStartSLOduration=3.962617236 podStartE2EDuration="22.894367762s" totalImagesPullingTime="18.931750526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:27.893877273 +0000 UTC m=+23.788064545" watchObservedRunningTime="2026-09-16 18:40:27.894367762 +0000 UTC m=+23.788555031"
Sep 16 18:40:28.025219 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:28.025178    2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:40:28.563288 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:28.563124    2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:40:28.025211594Z","UUID":"8c8c88f2-db85-488f-ad21-aca098777955","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:40:28.564829 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:28.564808    2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:40:28.564972 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:28.564843    2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:40:28.619365 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:28.619324    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:28.619600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:28.619340    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:28.619600 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:28.619469    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:28.619600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:28.619346    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:28.619600 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:28.619555    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:28.619786 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:28.619639    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:28.859426 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:28.859328    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jzjwf" event={"ID":"55e5f454-d5af-40ab-a53f-b0e12313348b","Type":"ContainerStarted","Data":"624cf5d5be5d89407b3e5197fec64ffc6f35d9acf33f4e1c7c364f041fba859d"}
Sep 16 18:40:28.862505 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:28.862326    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd" event={"ID":"2b943eaf-4b17-4d8b-9968-a5be25c2c678","Type":"ContainerStarted","Data":"29ac5db8cadbe25038a66b64dc5210e5d53eb9bb1730d65ba7f27e8e26295b9a"}
Sep 16 18:40:28.879857 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:28.879788    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jzjwf" podStartSLOduration=4.933387145 podStartE2EDuration="23.879761741s" totalImagesPullingTime="18.946374596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:28.879227573 +0000 UTC m=+24.773414845" watchObservedRunningTime="2026-09-16 18:40:28.879761741 +0000 UTC m=+24.773949011"
Sep 16 18:40:29.868037 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:29.867944    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd" event={"ID":"2b943eaf-4b17-4d8b-9968-a5be25c2c678","Type":"ContainerStarted","Data":"7ac11734309f281c0848d924279ff51820f5a856054d795f485a687d653a959c"}
Sep 16 18:40:29.872194 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:29.872162    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv" event={"ID":"db04d8f0-efe5-4d33-871b-8a2421f26e17","Type":"ContainerStarted","Data":"021dd8ef5e9fc532b22feda3b4813cff199d56feebd4cf5d19c8f9ce33a9786f"}
Sep 16 18:40:29.890613 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:29.890548    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2bbd" podStartSLOduration=3.7634442310000003 podStartE2EDuration="24.890522754s" totalImagesPullingTime="21.127078523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:29.889857036 +0000 UTC m=+25.784044305" watchObservedRunningTime="2026-09-16 18:40:29.890522754 +0000 UTC m=+25.784710026"
Sep 16 18:40:30.074766 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:30.074720    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9phx7"
Sep 16 18:40:30.075472 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:30.075440    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9phx7"
Sep 16 18:40:30.620108 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:30.620057    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:30.620304 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:30.620057    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:30.620304 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:30.620203    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:30.620304 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:30.620057    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:30.620304 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:30.620279    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:30.620494 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:30.620329    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:30.875627 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:30.875594    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9phx7"
Sep 16 18:40:30.876228 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:30.876182    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9phx7"
Sep 16 18:40:32.619978 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:32.619716    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:32.620488 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:32.619719    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:32.620488 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:32.619719    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:32.620488 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:32.620131    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:32.620488 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:32.620198    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:32.620488 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:32.620269    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:32.880644 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:32.880611    2552 generic.go:356] "Generic (PLEG): container finished" podID="346b4c37-63f2-42b6-989f-14bcff205a35" containerID="e52c4f6bab3e9e787637945f3aac9e2ad68896bff5b8fa22b84bf35e07448e85" exitCode=0
Sep 16 18:40:32.880844 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:32.880689    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnccg" event={"ID":"346b4c37-63f2-42b6-989f-14bcff205a35","Type":"ContainerDied","Data":"e52c4f6bab3e9e787637945f3aac9e2ad68896bff5b8fa22b84bf35e07448e85"}
Sep 16 18:40:32.884565 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:32.884533    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv" event={"ID":"db04d8f0-efe5-4d33-871b-8a2421f26e17","Type":"ContainerStarted","Data":"edf86fb3850a24cc04e4d8ca5f4092d16e0ebca0c8aca3c203c193f547f581ae"}
Sep 16 18:40:32.947007 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:32.946949    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv" podStartSLOduration=8.943480609 podStartE2EDuration="27.946930445s" totalImagesPullingTime="19.003449836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:32.946569141 +0000 UTC m=+28.840756410" watchObservedRunningTime="2026-09-16 18:40:32.946930445 +0000 UTC m=+28.841117691"
Sep 16 18:40:33.887250 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:33.887216    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:33.887964 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:33.887305    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:33.887964 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:33.887353    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:33.904463 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:33.904436    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:33.904675 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:33.904661    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:40:34.035534 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:34.035493    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g5lz9"]
Sep 16 18:40:34.035687 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:34.035666    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:34.035804 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:34.035778    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:34.037567 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:34.037533    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zkgpv"]
Sep 16 18:40:34.037713 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:34.037655    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:34.037768 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:34.037750    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:34.043743 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:34.043661    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qbtmz"]
Sep 16 18:40:34.043951 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:34.043838    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:34.044035 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:34.043940    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:34.894456 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:34.894422    2552 generic.go:356] "Generic (PLEG): container finished" podID="346b4c37-63f2-42b6-989f-14bcff205a35" containerID="a06a2f8266c97f405d0005424ed2802775ab926f1a235b208b7b1732db649552" exitCode=0
Sep 16 18:40:34.894812 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:34.894492    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnccg" event={"ID":"346b4c37-63f2-42b6-989f-14bcff205a35","Type":"ContainerDied","Data":"a06a2f8266c97f405d0005424ed2802775ab926f1a235b208b7b1732db649552"}
Sep 16 18:40:35.619875 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:35.619825    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:35.619875 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:35.619872    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:35.620170 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:35.619843    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:35.620170 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:35.619969    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:35.620170 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:35.620058    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:35.620275 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:35.620180    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:36.901968 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:36.901919    2552 generic.go:356] "Generic (PLEG): container finished" podID="346b4c37-63f2-42b6-989f-14bcff205a35" containerID="362b9bf656844df89a250726073df7c887568996c3ab3828a9200fc1d89abd17" exitCode=0
Sep 16 18:40:36.902426 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:36.901988    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnccg" event={"ID":"346b4c37-63f2-42b6-989f-14bcff205a35","Type":"ContainerDied","Data":"362b9bf656844df89a250726073df7c887568996c3ab3828a9200fc1d89abd17"}
Sep 16 18:40:37.620190 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:37.620152    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:37.620190 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:37.620177    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:37.620427 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:37.620180    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:37.620427 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:37.620285    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5lz9" podUID="8058be60-2358-4acd-bc5b-6382f4fa2e3e"
Sep 16 18:40:37.620427 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:37.620373    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbtmz" podUID="18bae23f-10bf-41df-a14b-ba453b1ab8f1"
Sep 16 18:40:37.620580 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:37.620466    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkgpv" podUID="0f5f1c97-1813-4d89-8af7-ce7d0eebc10c"
Sep 16 18:40:38.265373 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:38.265329    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs\") pod \"network-metrics-daemon-zkgpv\" (UID: \"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c\") " pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:38.265817 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:38.265501    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:38.265817 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:38.265576    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs podName:0f5f1c97-1813-4d89-8af7-ce7d0eebc10c nodeName:}" failed. No retries permitted until 2026-09-16 18:41:10.265559882 +0000 UTC m=+66.159747147 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs") pod "network-metrics-daemon-zkgpv" (UID: "0f5f1c97-1813-4d89-8af7-ce7d0eebc10c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:38.366147 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:38.366104    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdx4\" (UniqueName: \"kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4\") pod \"network-check-target-qbtmz\" (UID: \"18bae23f-10bf-41df-a14b-ba453b1ab8f1\") " pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:38.366346 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:38.366164    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:38.366346 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:38.366284    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:38.366346 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:38.366309    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:38.366346 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:38.366338    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:38.366658 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:38.366351    2552 projected.go:196] Error preparing data for projected volume kube-api-access-hvdx4 for pod openshift-network-diagnostics/network-check-target-qbtmz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:38.366658 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:38.366352    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret podName:8058be60-2358-4acd-bc5b-6382f4fa2e3e nodeName:}" failed. No retries permitted until 2026-09-16 18:41:10.366331481 +0000 UTC m=+66.260518728 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret") pod "global-pull-secret-syncer-g5lz9" (UID: "8058be60-2358-4acd-bc5b-6382f4fa2e3e") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:38.366658 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:38.366439    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4 podName:18bae23f-10bf-41df-a14b-ba453b1ab8f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:10.366423124 +0000 UTC m=+66.260610371 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvdx4" (UniqueName: "kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4") pod "network-check-target-qbtmz" (UID: "18bae23f-10bf-41df-a14b-ba453b1ab8f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:39.348164 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.348133    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-72.ec2.internal" event="NodeReady"
Sep 16 18:40:39.348593 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.348275    2552 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:40:39.384963 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.384924    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-886957dbf-7nb9z"]
Sep 16 18:40:39.387072 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.387044    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.389481 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.389402    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:40:39.389481 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.389409    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jkf2p\""
Sep 16 18:40:39.389676 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.389579    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:40:39.389747 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.389728    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:40:39.411957 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.411925    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bhkzx"]
Sep 16 18:40:39.413760 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.413733    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bhkzx"
Sep 16 18:40:39.416747 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.416717    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nkrcn\""
Sep 16 18:40:39.417764 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.417741    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:40:39.418141 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.418119    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:40:39.418308 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.418287    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:40:39.425195 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.425166    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cs4tf"]
Sep 16 18:40:39.427569 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.427543    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:40:39.427823 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.427796    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-886957dbf-7nb9z"]
Sep 16 18:40:39.427956 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.427924    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:39.431403 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.431380    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f92fp\""
Sep 16 18:40:39.438885 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.438856    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:40:39.439050 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.438926    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:40:39.445936 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.445842    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bhkzx"]
Sep 16 18:40:39.455368 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.455321    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cs4tf"]
Sep 16 18:40:39.472937 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.472899    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgfrz\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-kube-api-access-cgfrz\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.473145 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.472949    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.473145 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.472969    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/490ebb65-e892-4133-8653-fe085b6f034e-ca-trust-extracted\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.473145 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.473001    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-bound-sa-token\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.473145 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.473029    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/490ebb65-e892-4133-8653-fe085b6f034e-trusted-ca\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.473145 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.473132    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/490ebb65-e892-4133-8653-fe085b6f034e-image-registry-private-configuration\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.473381 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.473193    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/490ebb65-e892-4133-8653-fe085b6f034e-registry-certificates\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.473381 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.473226    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/490ebb65-e892-4133-8653-fe085b6f034e-installation-pull-secrets\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.574328 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.574292    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgfrz\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-kube-api-access-cgfrz\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.574328 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.574332    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:39.574584 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.574361    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzhq7\" (UniqueName: \"kubernetes.io/projected/083ee546-65d4-43b1-9cb4-383989984db0-kube-api-access-wzhq7\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:39.574584 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.574427    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.574584 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.574466    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/490ebb65-e892-4133-8653-fe085b6f034e-ca-trust-extracted\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.574584 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.574485    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-bound-sa-token\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.574584 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.574507    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/083ee546-65d4-43b1-9cb4-383989984db0-config-volume\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:39.574584 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.574537    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/490ebb65-e892-4133-8653-fe085b6f034e-trusted-ca\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.574584 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:39.574570    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:39.574917 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:39.574598    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-886957dbf-7nb9z: secret "image-registry-tls" not found
Sep 16 18:40:39.574917 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.574578    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f31a781-91cf-40f7-94e7-6560c4d94cfa-cert\") pod \"ingress-canary-bhkzx\" (UID: \"0f31a781-91cf-40f7-94e7-6560c4d94cfa\") " pod="openshift-ingress-canary/ingress-canary-bhkzx"
Sep 16 18:40:39.574917 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:39.574655    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls podName:490ebb65-e892-4133-8653-fe085b6f034e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:40.074635545 +0000 UTC m=+35.968822792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls") pod "image-registry-886957dbf-7nb9z" (UID: "490ebb65-e892-4133-8653-fe085b6f034e") : secret "image-registry-tls" not found
Sep 16 18:40:39.574917 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.574696    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/490ebb65-e892-4133-8653-fe085b6f034e-image-registry-private-configuration\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.574917 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.574748    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/490ebb65-e892-4133-8653-fe085b6f034e-registry-certificates\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.574917 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.574774    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/490ebb65-e892-4133-8653-fe085b6f034e-installation-pull-secrets\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.574917 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.574801    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djgh6\" (UniqueName: \"kubernetes.io/projected/0f31a781-91cf-40f7-94e7-6560c4d94cfa-kube-api-access-djgh6\") pod \"ingress-canary-bhkzx\" (UID: \"0f31a781-91cf-40f7-94e7-6560c4d94cfa\") " pod="openshift-ingress-canary/ingress-canary-bhkzx"
Sep 16 18:40:39.574917 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.574833    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083ee546-65d4-43b1-9cb4-383989984db0-tmp-dir\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:39.575255 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.574931    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/490ebb65-e892-4133-8653-fe085b6f034e-ca-trust-extracted\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.575357 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.575317    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/490ebb65-e892-4133-8653-fe085b6f034e-registry-certificates\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.575586 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.575567    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/490ebb65-e892-4133-8653-fe085b6f034e-trusted-ca\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.580959 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.580798    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/490ebb65-e892-4133-8653-fe085b6f034e-installation-pull-secrets\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.581128 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.580800    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/490ebb65-e892-4133-8653-fe085b6f034e-image-registry-private-configuration\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.587406 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.587371    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgfrz\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-kube-api-access-cgfrz\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.588877 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.588851    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-bound-sa-token\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:39.620015 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.619974    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:40:39.620294 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.619974    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:40:39.620294 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.619974    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:40:39.622692 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.622665    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:40:39.623080 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.623051    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:40:39.623376 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.623358    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-md6w5\""
Sep 16 18:40:39.623376 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.623368    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:40:39.623496 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.623386    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mffp6\""
Sep 16 18:40:39.623684 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.623657    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:40:39.675638 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.675550    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083ee546-65d4-43b1-9cb4-383989984db0-tmp-dir\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:39.675638 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.675607    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:39.675638 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.675640    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhq7\" (UniqueName: \"kubernetes.io/projected/083ee546-65d4-43b1-9cb4-383989984db0-kube-api-access-wzhq7\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:39.675915 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.675689    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/083ee546-65d4-43b1-9cb4-383989984db0-config-volume\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:39.675915 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:39.675808    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:39.675915 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.675839    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f31a781-91cf-40f7-94e7-6560c4d94cfa-cert\") pod \"ingress-canary-bhkzx\" (UID: \"0f31a781-91cf-40f7-94e7-6560c4d94cfa\") " pod="openshift-ingress-canary/ingress-canary-bhkzx"
Sep 16 18:40:39.675915 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:39.675883    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls podName:083ee546-65d4-43b1-9cb4-383989984db0 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:40.17586307 +0000 UTC m=+36.070050321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls") pod "dns-default-cs4tf" (UID: "083ee546-65d4-43b1-9cb4-383989984db0") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:39.676141 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.675944    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djgh6\" (UniqueName: \"kubernetes.io/projected/0f31a781-91cf-40f7-94e7-6560c4d94cfa-kube-api-access-djgh6\") pod \"ingress-canary-bhkzx\" (UID: \"0f31a781-91cf-40f7-94e7-6560c4d94cfa\") " pod="openshift-ingress-canary/ingress-canary-bhkzx"
Sep 16 18:40:39.676141 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.675964    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083ee546-65d4-43b1-9cb4-383989984db0-tmp-dir\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:39.676415 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.676387    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/083ee546-65d4-43b1-9cb4-383989984db0-config-volume\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:39.678683 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.678656    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f31a781-91cf-40f7-94e7-6560c4d94cfa-cert\") pod \"ingress-canary-bhkzx\" (UID: \"0f31a781-91cf-40f7-94e7-6560c4d94cfa\") " pod="openshift-ingress-canary/ingress-canary-bhkzx"
Sep 16 18:40:39.691744 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.691700    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhq7\" (UniqueName: \"kubernetes.io/projected/083ee546-65d4-43b1-9cb4-383989984db0-kube-api-access-wzhq7\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:39.691913 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.691771    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djgh6\" (UniqueName: \"kubernetes.io/projected/0f31a781-91cf-40f7-94e7-6560c4d94cfa-kube-api-access-djgh6\") pod \"ingress-canary-bhkzx\" (UID: \"0f31a781-91cf-40f7-94e7-6560c4d94cfa\") " pod="openshift-ingress-canary/ingress-canary-bhkzx"
Sep 16 18:40:39.724723 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:39.724684    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bhkzx"
Sep 16 18:40:40.080345 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:40.080260    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:40.080514 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:40.080439    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:40.080514 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:40.080458    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-886957dbf-7nb9z: secret "image-registry-tls" not found
Sep 16 18:40:40.080624 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:40.080520    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls podName:490ebb65-e892-4133-8653-fe085b6f034e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:41.080501008 +0000 UTC m=+36.974688258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls") pod "image-registry-886957dbf-7nb9z" (UID: "490ebb65-e892-4133-8653-fe085b6f034e") : secret "image-registry-tls" not found
Sep 16 18:40:40.181494 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:40.181457    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:40.181687 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:40.181602    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:40.181748 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:40.181700    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls podName:083ee546-65d4-43b1-9cb4-383989984db0 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:41.181681123 +0000 UTC m=+37.075868381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls") pod "dns-default-cs4tf" (UID: "083ee546-65d4-43b1-9cb4-383989984db0") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:40.915219 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:40.915188    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnccg" event={"ID":"346b4c37-63f2-42b6-989f-14bcff205a35","Type":"ContainerStarted","Data":"b02f7109002ec913bfd9be91432dca5737ee84e6aa092244c4da75b67c0e5a47"}
Sep 16 18:40:41.057395 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:41.057312    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bhkzx"]
Sep 16 18:40:41.060657 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:40:41.060620    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f31a781_91cf_40f7_94e7_6560c4d94cfa.slice/crio-1747a87107680aeecdca49fbccde7730136918cbba6f7fc29413f2c30b18b77a WatchSource:0}: Error finding container 1747a87107680aeecdca49fbccde7730136918cbba6f7fc29413f2c30b18b77a: Status 404 returned error can't find the container with id 1747a87107680aeecdca49fbccde7730136918cbba6f7fc29413f2c30b18b77a
Sep 16 18:40:41.089418 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:41.089378    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:41.089581 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:41.089553    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:41.089740 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:41.089585    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-886957dbf-7nb9z: secret "image-registry-tls" not found
Sep 16 18:40:41.089740 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:41.089648    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls podName:490ebb65-e892-4133-8653-fe085b6f034e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:43.089629045 +0000 UTC m=+38.983816316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls") pod "image-registry-886957dbf-7nb9z" (UID: "490ebb65-e892-4133-8653-fe085b6f034e") : secret "image-registry-tls" not found
Sep 16 18:40:41.190648 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:41.190594    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:41.190833 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:41.190756    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:41.190873 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:41.190840    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls podName:083ee546-65d4-43b1-9cb4-383989984db0 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:43.190822699 +0000 UTC m=+39.085009946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls") pod "dns-default-cs4tf" (UID: "083ee546-65d4-43b1-9cb4-383989984db0") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:41.919537 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:41.919503    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bhkzx" event={"ID":"0f31a781-91cf-40f7-94e7-6560c4d94cfa","Type":"ContainerStarted","Data":"1747a87107680aeecdca49fbccde7730136918cbba6f7fc29413f2c30b18b77a"}
Sep 16 18:40:41.922664 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:41.922637    2552 generic.go:356] "Generic (PLEG): container finished" podID="346b4c37-63f2-42b6-989f-14bcff205a35" containerID="b02f7109002ec913bfd9be91432dca5737ee84e6aa092244c4da75b67c0e5a47" exitCode=0
Sep 16 18:40:41.922792 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:41.922701    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnccg" event={"ID":"346b4c37-63f2-42b6-989f-14bcff205a35","Type":"ContainerDied","Data":"b02f7109002ec913bfd9be91432dca5737ee84e6aa092244c4da75b67c0e5a47"}
Sep 16 18:40:42.929554 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:42.929518    2552 generic.go:356] "Generic (PLEG): container finished" podID="346b4c37-63f2-42b6-989f-14bcff205a35" containerID="b25431ef47b32e4e3b625fe10f26b0b34c27dc158a1cc3efd11673d8fcd9b265" exitCode=0
Sep 16 18:40:42.929986 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:42.929588    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnccg" event={"ID":"346b4c37-63f2-42b6-989f-14bcff205a35","Type":"ContainerDied","Data":"b25431ef47b32e4e3b625fe10f26b0b34c27dc158a1cc3efd11673d8fcd9b265"}
Sep 16 18:40:43.108756 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:43.108521    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:43.108854 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:43.108664    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:43.108854 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:43.108787    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-886957dbf-7nb9z: secret "image-registry-tls" not found
Sep 16 18:40:43.108854 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:43.108850    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls podName:490ebb65-e892-4133-8653-fe085b6f034e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:47.108832034 +0000 UTC m=+43.003019295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls") pod "image-registry-886957dbf-7nb9z" (UID: "490ebb65-e892-4133-8653-fe085b6f034e") : secret "image-registry-tls" not found
Sep 16 18:40:43.210227 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:43.210142    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:43.210318 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:43.210279    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:43.210361 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:43.210333    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls podName:083ee546-65d4-43b1-9cb4-383989984db0 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:47.210318901 +0000 UTC m=+43.104506149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls") pod "dns-default-cs4tf" (UID: "083ee546-65d4-43b1-9cb4-383989984db0") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:43.935041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:43.935013    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnccg" event={"ID":"346b4c37-63f2-42b6-989f-14bcff205a35","Type":"ContainerStarted","Data":"a9ff2d2bb8576b8eb8016b182a878c27983fec8e62522ebfbabfaf200e7cf70f"}
Sep 16 18:40:43.936416 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:43.936393    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bhkzx" event={"ID":"0f31a781-91cf-40f7-94e7-6560c4d94cfa","Type":"ContainerStarted","Data":"7f92a3bf02c026111b5ccb80768e194ab6badfd3d09a2bb497e0c3ebccae947f"}
Sep 16 18:40:43.970029 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:43.969982    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rnccg" podStartSLOduration=9.682098826 podStartE2EDuration="38.969967502s" totalImagesPullingTime="29.287868676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:43.969322507 +0000 UTC m=+39.863509789" watchObservedRunningTime="2026-09-16 18:40:43.969967502 +0000 UTC m=+39.864154771"
Sep 16 18:40:43.992112 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:43.991196    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bhkzx" podStartSLOduration=2.767151266 podStartE2EDuration="4.9911773s" totalImagesPullingTime="2.224026034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:43.989939214 +0000 UTC m=+39.884126484" watchObservedRunningTime="2026-09-16 18:40:43.9911773 +0000 UTC m=+39.885364571"
Sep 16 18:40:47.139157 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:47.139113    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:47.139544 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:47.139279    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:47.139544 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:47.139303    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-886957dbf-7nb9z: secret "image-registry-tls" not found
Sep 16 18:40:47.139544 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:47.139360    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls podName:490ebb65-e892-4133-8653-fe085b6f034e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:55.139345043 +0000 UTC m=+51.033532290 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls") pod "image-registry-886957dbf-7nb9z" (UID: "490ebb65-e892-4133-8653-fe085b6f034e") : secret "image-registry-tls" not found
Sep 16 18:40:47.239512 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:47.239468    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:47.239675 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:47.239624    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:47.239711 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:47.239689    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls podName:083ee546-65d4-43b1-9cb4-383989984db0 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:55.239672267 +0000 UTC m=+51.133859514 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls") pod "dns-default-cs4tf" (UID: "083ee546-65d4-43b1-9cb4-383989984db0") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:55.194345 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:55.194294    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:40:55.194795 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:55.194435    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:55.194795 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:55.194455    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-886957dbf-7nb9z: secret "image-registry-tls" not found
Sep 16 18:40:55.194795 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:55.194512    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls podName:490ebb65-e892-4133-8653-fe085b6f034e nodeName:}" failed. No retries permitted until 2026-09-16 18:41:11.194496349 +0000 UTC m=+67.088683596 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls") pod "image-registry-886957dbf-7nb9z" (UID: "490ebb65-e892-4133-8653-fe085b6f034e") : secret "image-registry-tls" not found
Sep 16 18:40:55.294849 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:40:55.294811    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:40:55.294987 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:55.294923    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:55.294987 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:40:55.294983    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls podName:083ee546-65d4-43b1-9cb4-383989984db0 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:11.294970714 +0000 UTC m=+67.189157961 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls") pod "dns-default-cs4tf" (UID: "083ee546-65d4-43b1-9cb4-383989984db0") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:05.908046 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:05.908011    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cs8zv"
Sep 16 18:41:10.303820 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:10.303760    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs\") pod \"network-metrics-daemon-zkgpv\" (UID: \"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c\") " pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:41:10.306322 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:10.306297    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:10.314814 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:10.314780    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:41:10.314907 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:10.314895    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs podName:0f5f1c97-1813-4d89-8af7-ce7d0eebc10c nodeName:}" failed. No retries permitted until 2026-09-16 18:42:14.314876609 +0000 UTC m=+130.209063856 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs") pod "network-metrics-daemon-zkgpv" (UID: "0f5f1c97-1813-4d89-8af7-ce7d0eebc10c") : secret "metrics-daemon-secret" not found
Sep 16 18:41:10.404764 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:10.404725    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdx4\" (UniqueName: \"kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4\") pod \"network-check-target-qbtmz\" (UID: \"18bae23f-10bf-41df-a14b-ba453b1ab8f1\") " pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:41:10.404835 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:10.404775    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:41:10.407516 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:10.407479    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:10.407665 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:10.407528    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:10.418525 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:10.418497    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:10.419587 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:10.419561    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8058be60-2358-4acd-bc5b-6382f4fa2e3e-original-pull-secret\") pod \"global-pull-secret-syncer-g5lz9\" (UID: \"8058be60-2358-4acd-bc5b-6382f4fa2e3e\") " pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:41:10.428860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:10.428828    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvdx4\" (UniqueName: \"kubernetes.io/projected/18bae23f-10bf-41df-a14b-ba453b1ab8f1-kube-api-access-hvdx4\") pod \"network-check-target-qbtmz\" (UID: \"18bae23f-10bf-41df-a14b-ba453b1ab8f1\") " pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:41:10.535785 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:10.535748    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mffp6\""
Sep 16 18:41:10.540074 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:10.540044    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5lz9"
Sep 16 18:41:10.544192 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:10.544163    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:41:10.826420 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:10.826347    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g5lz9"]
Sep 16 18:41:10.829365 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:41:10.829335    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8058be60_2358_4acd_bc5b_6382f4fa2e3e.slice/crio-f8548048c1a3cdec2b643aa3a557876a1b7a26db572b8f3d246e7a8228f8aa62 WatchSource:0}: Error finding container f8548048c1a3cdec2b643aa3a557876a1b7a26db572b8f3d246e7a8228f8aa62: Status 404 returned error can't find the container with id f8548048c1a3cdec2b643aa3a557876a1b7a26db572b8f3d246e7a8228f8aa62
Sep 16 18:41:10.835886 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:10.835858    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qbtmz"]
Sep 16 18:41:10.839147 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:41:10.839122    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bae23f_10bf_41df_a14b_ba453b1ab8f1.slice/crio-4b58a055149d218bc286b40dcbc61d8ade3d821c4fa5a4dfc4192c47d7bb4796 WatchSource:0}: Error finding container 4b58a055149d218bc286b40dcbc61d8ade3d821c4fa5a4dfc4192c47d7bb4796: Status 404 returned error can't find the container with id 4b58a055149d218bc286b40dcbc61d8ade3d821c4fa5a4dfc4192c47d7bb4796
Sep 16 18:41:11.001950 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:11.001910    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g5lz9" event={"ID":"8058be60-2358-4acd-bc5b-6382f4fa2e3e","Type":"ContainerStarted","Data":"f8548048c1a3cdec2b643aa3a557876a1b7a26db572b8f3d246e7a8228f8aa62"}
Sep 16 18:41:11.003129 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:11.003104    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qbtmz" event={"ID":"18bae23f-10bf-41df-a14b-ba453b1ab8f1","Type":"ContainerStarted","Data":"4b58a055149d218bc286b40dcbc61d8ade3d821c4fa5a4dfc4192c47d7bb4796"}
Sep 16 18:41:11.210609 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:11.210562    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:41:11.210792 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:11.210706    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:11.210792 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:11.210722    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-886957dbf-7nb9z: secret "image-registry-tls" not found
Sep 16 18:41:11.210792 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:11.210775    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls podName:490ebb65-e892-4133-8653-fe085b6f034e nodeName:}" failed. No retries permitted until 2026-09-16 18:41:43.210760363 +0000 UTC m=+99.104947615 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls") pod "image-registry-886957dbf-7nb9z" (UID: "490ebb65-e892-4133-8653-fe085b6f034e") : secret "image-registry-tls" not found
Sep 16 18:41:11.311294 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:11.311253    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:41:11.311787 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:11.311432    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:11.311787 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:11.311523    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls podName:083ee546-65d4-43b1-9cb4-383989984db0 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:43.311495009 +0000 UTC m=+99.205682259 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls") pod "dns-default-cs4tf" (UID: "083ee546-65d4-43b1-9cb4-383989984db0") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:16.019287 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:16.019248    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qbtmz" event={"ID":"18bae23f-10bf-41df-a14b-ba453b1ab8f1","Type":"ContainerStarted","Data":"c47c0fb9b07b72565aa2e299ff19fa42f7b0f018705678ffd6fd585cf576700c"}
Sep 16 18:41:16.019868 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:16.019360    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:41:16.020881 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:16.020856    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g5lz9" event={"ID":"8058be60-2358-4acd-bc5b-6382f4fa2e3e","Type":"ContainerStarted","Data":"1c077f57c6b8c9090aa70bb9291076650a08022de020e62ca50e51b644907f09"}
Sep 16 18:41:16.039677 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:16.039618    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qbtmz" podStartSLOduration=66.661574789 podStartE2EDuration="1m11.039600795s" totalImagesPullingTime="4.378026006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:16.039023315 +0000 UTC m=+71.933210586" watchObservedRunningTime="2026-09-16 18:41:16.039600795 +0000 UTC m=+71.933788064"
Sep 16 18:41:16.056667 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:16.056611    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g5lz9" podStartSLOduration=65.66537664 podStartE2EDuration="1m10.056593757s" totalImagesPullingTime="4.391217117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:16.056500405 +0000 UTC m=+71.950687695" watchObservedRunningTime="2026-09-16 18:41:16.056593757 +0000 UTC m=+71.950781025"
Sep 16 18:41:37.215159 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.214979    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-bd67b9f94-9bddk"]
Sep 16 18:41:37.219285 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.219260    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.221624 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.221597    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:41:37.222212 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.222192    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 18:41:37.222286 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.222265    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 18:41:37.222512 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.222493    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:41:37.222632 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.222493    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 18:41:37.222632 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.222499    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 18:41:37.223346 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.223327    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p4t2n\""
Sep 16 18:41:37.241877 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.241844    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-bd67b9f94-9bddk"]
Sep 16 18:41:37.311608 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.311571    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-stats-auth\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.311608 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.311623    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.311821 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.311690    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-default-certificate\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.311821 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.311722    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92jr7\" (UniqueName: \"kubernetes.io/projected/d9a7b520-0ba2-428b-9927-d671e633215a-kube-api-access-92jr7\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.311821 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.311747    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.412173 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.412120    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.412173 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.412178    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-default-certificate\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.412428 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.412200    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92jr7\" (UniqueName: \"kubernetes.io/projected/d9a7b520-0ba2-428b-9927-d671e633215a-kube-api-access-92jr7\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.412428 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.412220    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.412428 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:37.412297    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle podName:d9a7b520-0ba2-428b-9927-d671e633215a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.912277507 +0000 UTC m=+93.806464754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle") pod "router-default-bd67b9f94-9bddk" (UID: "d9a7b520-0ba2-428b-9927-d671e633215a") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:37.412428 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.412337    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-stats-auth\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.412428 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:37.412413    2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:41:37.412671 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:37.412475    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs podName:d9a7b520-0ba2-428b-9927-d671e633215a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.912458327 +0000 UTC m=+93.806645594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs") pod "router-default-bd67b9f94-9bddk" (UID: "d9a7b520-0ba2-428b-9927-d671e633215a") : secret "router-metrics-certs-default" not found
Sep 16 18:41:37.414818 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.414790    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-default-certificate\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.414947 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.414925    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-stats-auth\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.466831 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.466751    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92jr7\" (UniqueName: \"kubernetes.io/projected/d9a7b520-0ba2-428b-9927-d671e633215a-kube-api-access-92jr7\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.917251 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.917214    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.917467 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:37.917263    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:37.917467 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:37.917437    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle podName:d9a7b520-0ba2-428b-9927-d671e633215a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:38.917412354 +0000 UTC m=+94.811599615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle") pod "router-default-bd67b9f94-9bddk" (UID: "d9a7b520-0ba2-428b-9927-d671e633215a") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:37.917599 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:37.917471    2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:41:37.917599 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:37.917512    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs podName:d9a7b520-0ba2-428b-9927-d671e633215a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:38.917499733 +0000 UTC m=+94.811686979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs") pod "router-default-bd67b9f94-9bddk" (UID: "d9a7b520-0ba2-428b-9927-d671e633215a") : secret "router-metrics-certs-default" not found
Sep 16 18:41:38.924777 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:38.924732    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:38.925196 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:38.924786    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:38.925196 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:38.924892    2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:41:38.925196 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:38.924918    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle podName:d9a7b520-0ba2-428b-9927-d671e633215a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:40.924898108 +0000 UTC m=+96.819085383 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle") pod "router-default-bd67b9f94-9bddk" (UID: "d9a7b520-0ba2-428b-9927-d671e633215a") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:38.925196 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:38.924944    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs podName:d9a7b520-0ba2-428b-9927-d671e633215a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:40.924932168 +0000 UTC m=+96.819119419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs") pod "router-default-bd67b9f94-9bddk" (UID: "d9a7b520-0ba2-428b-9927-d671e633215a") : secret "router-metrics-certs-default" not found
Sep 16 18:41:40.940352 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:40.940297    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:40.940800 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:40.940414    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:40.940800 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:40.940463    2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:41:40.940800 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:40.940541    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs podName:d9a7b520-0ba2-428b-9927-d671e633215a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:44.940524141 +0000 UTC m=+100.834711392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs") pod "router-default-bd67b9f94-9bddk" (UID: "d9a7b520-0ba2-428b-9927-d671e633215a") : secret "router-metrics-certs-default" not found
Sep 16 18:41:40.940800 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:40.940556    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle podName:d9a7b520-0ba2-428b-9927-d671e633215a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:44.940549875 +0000 UTC m=+100.834737121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle") pod "router-default-bd67b9f94-9bddk" (UID: "d9a7b520-0ba2-428b-9927-d671e633215a") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:41.413760 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.413701    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f8z6x"]
Sep 16 18:41:41.417699 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.417668    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:41.420251 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.420221    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:41:41.421155 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.421135    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:41:41.421320 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.421303    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:41:41.421384 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.421372    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8f7h6\""
Sep 16 18:41:41.423821 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.423797    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:41:41.444998 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.444957    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f8z6x"]
Sep 16 18:41:41.546211 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.546172    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52kht\" (UniqueName: \"kubernetes.io/projected/cc51025e-e317-4422-8196-00e87137b6f1-kube-api-access-52kht\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:41.546404 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.546236    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc51025e-e317-4422-8196-00e87137b6f1-crio-socket\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:41.546404 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.546257    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:41.546404 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.546306    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc51025e-e317-4422-8196-00e87137b6f1-data-volume\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:41.546404 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.546353    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:41.647356 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.647311    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc51025e-e317-4422-8196-00e87137b6f1-data-volume\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:41.647356 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.647360    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:41.647555 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:41.647509    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:41.647555 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.647541    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52kht\" (UniqueName: \"kubernetes.io/projected/cc51025e-e317-4422-8196-00e87137b6f1-kube-api-access-52kht\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:41.647618 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:41.647603    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm podName:cc51025e-e317-4422-8196-00e87137b6f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:42.147579109 +0000 UTC m=+98.041766357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8z6x" (UID: "cc51025e-e317-4422-8196-00e87137b6f1") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:41.647664 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.647627    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc51025e-e317-4422-8196-00e87137b6f1-crio-socket\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:41.647698 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.647660    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:41.647732 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.647707    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc51025e-e317-4422-8196-00e87137b6f1-data-volume\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:41.647775 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.647749    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc51025e-e317-4422-8196-00e87137b6f1-crio-socket\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:41.647810 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:41.647756    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:41.647845 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:41.647829    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls podName:cc51025e-e317-4422-8196-00e87137b6f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:42.147817859 +0000 UTC m=+98.042005105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8z6x" (UID: "cc51025e-e317-4422-8196-00e87137b6f1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:41.660938 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:41.660895    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52kht\" (UniqueName: \"kubernetes.io/projected/cc51025e-e317-4422-8196-00e87137b6f1-kube-api-access-52kht\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:42.151294 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:42.151254    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:42.151683 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:42.151323    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:42.151683 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:42.151381    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:42.151683 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:42.151441    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:42.151683 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:42.151481    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm podName:cc51025e-e317-4422-8196-00e87137b6f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:43.151465007 +0000 UTC m=+99.045652270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8z6x" (UID: "cc51025e-e317-4422-8196-00e87137b6f1") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:42.151683 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:42.151496    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls podName:cc51025e-e317-4422-8196-00e87137b6f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:43.151490004 +0000 UTC m=+99.045677251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8z6x" (UID: "cc51025e-e317-4422-8196-00e87137b6f1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:43.161053 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:43.161003    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:43.161470 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:43.161079    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:43.161470 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:43.161165    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:43.161470 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:43.161203    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:43.161470 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:43.161245    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm podName:cc51025e-e317-4422-8196-00e87137b6f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:45.161226099 +0000 UTC m=+101.055413352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8z6x" (UID: "cc51025e-e317-4422-8196-00e87137b6f1") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:43.161470 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:43.161261    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls podName:cc51025e-e317-4422-8196-00e87137b6f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:45.161254438 +0000 UTC m=+101.055441685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8z6x" (UID: "cc51025e-e317-4422-8196-00e87137b6f1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:43.262456 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:43.262418    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls\") pod \"image-registry-886957dbf-7nb9z\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") " pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:41:43.262547 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:43.262536    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:43.262583 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:43.262550    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-886957dbf-7nb9z: secret "image-registry-tls" not found
Sep 16 18:41:43.262615 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:43.262598    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls podName:490ebb65-e892-4133-8653-fe085b6f034e nodeName:}" failed. No retries permitted until 2026-09-16 18:42:47.262584594 +0000 UTC m=+163.156771841 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls") pod "image-registry-886957dbf-7nb9z" (UID: "490ebb65-e892-4133-8653-fe085b6f034e") : secret "image-registry-tls" not found
Sep 16 18:41:43.363289 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:43.363259    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:41:43.363436 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:43.363414    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:43.363489 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:43.363479    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls podName:083ee546-65d4-43b1-9cb4-383989984db0 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:47.363464637 +0000 UTC m=+163.257651887 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls") pod "dns-default-cs4tf" (UID: "083ee546-65d4-43b1-9cb4-383989984db0") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:43.377706 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:43.377679    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tw7bx_be0cf19b-10fb-41c7-be6b-efb9c89b566e/dns-node-resolver/0.log"
Sep 16 18:41:44.580003 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:44.579952    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z4qmj_aa40654a-058d-4aab-8eeb-b34d81ac854b/node-ca/0.log"
Sep 16 18:41:44.975424 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:44.975378    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:44.975612 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:44.975506    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:44.975612 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:44.975555    2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:41:44.975695 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:44.975629    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs podName:d9a7b520-0ba2-428b-9927-d671e633215a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:52.975607122 +0000 UTC m=+108.869794383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs") pod "router-default-bd67b9f94-9bddk" (UID: "d9a7b520-0ba2-428b-9927-d671e633215a") : secret "router-metrics-certs-default" not found
Sep 16 18:41:44.975695 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:44.975649    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle podName:d9a7b520-0ba2-428b-9927-d671e633215a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:52.975640163 +0000 UTC m=+108.869827428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle") pod "router-default-bd67b9f94-9bddk" (UID: "d9a7b520-0ba2-428b-9927-d671e633215a") : configmap references non-existent config key: service-ca.crt
Sep 16 18:41:45.177199 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:45.177153    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:45.177412 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:45.177257    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:45.177412 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:45.177316    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:45.177412 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:45.177361    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:45.177412 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:45.177387    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls podName:cc51025e-e317-4422-8196-00e87137b6f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:49.177370441 +0000 UTC m=+105.071557687 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8z6x" (UID: "cc51025e-e317-4422-8196-00e87137b6f1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:45.177412 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:45.177414    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm podName:cc51025e-e317-4422-8196-00e87137b6f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:49.177399137 +0000 UTC m=+105.071586383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8z6x" (UID: "cc51025e-e317-4422-8196-00e87137b6f1") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:45.194922 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:45.194891    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bhkzx_0f31a781-91cf-40f7-94e7-6560c4d94cfa/serve-healthcheck-canary/0.log"
Sep 16 18:41:47.026328 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:47.026288    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qbtmz"
Sep 16 18:41:49.211662 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:49.211627    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:49.212132 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:49.211697    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:49.212132 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:49.211777    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:49.212132 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:49.211837    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm podName:cc51025e-e317-4422-8196-00e87137b6f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:57.211822376 +0000 UTC m=+113.106009623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8z6x" (UID: "cc51025e-e317-4422-8196-00e87137b6f1") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:49.212132 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:49.211778    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:49.212132 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:41:49.211905    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls podName:cc51025e-e317-4422-8196-00e87137b6f1 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:57.211892036 +0000 UTC m=+113.106079287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8z6x" (UID: "cc51025e-e317-4422-8196-00e87137b6f1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:53.044706 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:53.044663    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:53.044706 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:53.044718    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:53.045481 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:53.045453    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9a7b520-0ba2-428b-9927-d671e633215a-service-ca-bundle\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:53.047081 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:53.047060    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a7b520-0ba2-428b-9927-d671e633215a-metrics-certs\") pod \"router-default-bd67b9f94-9bddk\" (UID: \"d9a7b520-0ba2-428b-9927-d671e633215a\") " pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:53.129148 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:53.129103    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:53.408928 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:53.408892    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-bd67b9f94-9bddk"]
Sep 16 18:41:53.412223 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:41:53.412197    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a7b520_0ba2_428b_9927_d671e633215a.slice/crio-d2089a98bfacf80d02daf5f17f1d0295d5ac1b96cfe81429646533c7e4c5a262 WatchSource:0}: Error finding container d2089a98bfacf80d02daf5f17f1d0295d5ac1b96cfe81429646533c7e4c5a262: Status 404 returned error can't find the container with id d2089a98bfacf80d02daf5f17f1d0295d5ac1b96cfe81429646533c7e4c5a262
Sep 16 18:41:54.115173 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:54.115136    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-bd67b9f94-9bddk" event={"ID":"d9a7b520-0ba2-428b-9927-d671e633215a","Type":"ContainerStarted","Data":"156b58244a9caef2e42bb637e94a760f71f6a110ea44b4362e660cb863da8d81"}
Sep 16 18:41:54.115656 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:54.115186    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-bd67b9f94-9bddk" event={"ID":"d9a7b520-0ba2-428b-9927-d671e633215a","Type":"ContainerStarted","Data":"d2089a98bfacf80d02daf5f17f1d0295d5ac1b96cfe81429646533c7e4c5a262"}
Sep 16 18:41:54.129694 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:54.129652    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:54.132432 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:54.132408    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:54.140848 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:54.140796    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-bd67b9f94-9bddk" podStartSLOduration=17.14078085 podStartE2EDuration="17.14078085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:54.140603292 +0000 UTC m=+110.034790572" watchObservedRunningTime="2026-09-16 18:41:54.14078085 +0000 UTC m=+110.034968132"
Sep 16 18:41:55.118147 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:55.118108    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:55.119445 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:55.119420    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-bd67b9f94-9bddk"
Sep 16 18:41:57.281365 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:57.281325    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:57.281835 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:57.281421    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:57.281956 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:57.281930    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc51025e-e317-4422-8196-00e87137b6f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:57.283808 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:57.283783    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc51025e-e317-4422-8196-00e87137b6f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8z6x\" (UID: \"cc51025e-e317-4422-8196-00e87137b6f1\") " pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:57.326860 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:57.326808    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f8z6x"
Sep 16 18:41:57.600783 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:57.600703    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f8z6x"]
Sep 16 18:41:57.603951 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:41:57.603914    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc51025e_e317_4422_8196_00e87137b6f1.slice/crio-ba5848fc299c18856b0dcf9000774cd039a675eb7ed2526db8f39ba10823fe50 WatchSource:0}: Error finding container ba5848fc299c18856b0dcf9000774cd039a675eb7ed2526db8f39ba10823fe50: Status 404 returned error can't find the container with id ba5848fc299c18856b0dcf9000774cd039a675eb7ed2526db8f39ba10823fe50
Sep 16 18:41:58.128447 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:58.128404    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8z6x" event={"ID":"cc51025e-e317-4422-8196-00e87137b6f1","Type":"ContainerStarted","Data":"b0dcc4f2db1c63774d46bb6d8b61bb12013ccd9f1a6bba7d45bfad5a47cfa43b"}
Sep 16 18:41:58.128630 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:58.128475    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8z6x" event={"ID":"cc51025e-e317-4422-8196-00e87137b6f1","Type":"ContainerStarted","Data":"ba5848fc299c18856b0dcf9000774cd039a675eb7ed2526db8f39ba10823fe50"}
Sep 16 18:41:59.134864 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:41:59.134823    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8z6x" event={"ID":"cc51025e-e317-4422-8196-00e87137b6f1","Type":"ContainerStarted","Data":"4e86c1077be8dd13bdc41d5033b16c11b11e281e5395ad60814e298465f02f59"}
Sep 16 18:42:00.139836 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:00.139795    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8z6x" event={"ID":"cc51025e-e317-4422-8196-00e87137b6f1","Type":"ContainerStarted","Data":"669d056460477b5ab876f53f981ec63dec825ecfc845bd5cfdfe2e4dd3a3555b"}
Sep 16 18:42:00.159692 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:00.159579    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f8z6x" podStartSLOduration=17.392545892 podStartE2EDuration="19.159554468s" totalImagesPullingTime="1.767008576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:41:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:00.15870085 +0000 UTC m=+116.052888125" watchObservedRunningTime="2026-09-16 18:42:00.159554468 +0000 UTC m=+116.053741740"
Sep 16 18:42:08.081313 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.081276    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7"]
Sep 16 18:42:08.083262 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.083233    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7"
Sep 16 18:42:08.087734 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.087703    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:42:08.087922 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.087900    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:42:08.088010 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.087903    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8czhz\""
Sep 16 18:42:08.088981 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.088958    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 18:42:08.089115 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.088999    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:42:08.098768 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.098732    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7"]
Sep 16 18:42:08.163041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.162996    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnlsd\" (UniqueName: \"kubernetes.io/projected/334663b8-eb31-4dd4-98ff-734383066552-kube-api-access-cnlsd\") pod \"managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7\" (UID: \"334663b8-eb31-4dd4-98ff-734383066552\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7"
Sep 16 18:42:08.163041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.163044    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/334663b8-eb31-4dd4-98ff-734383066552-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7\" (UID: \"334663b8-eb31-4dd4-98ff-734383066552\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7"
Sep 16 18:42:08.264404 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.264360    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlsd\" (UniqueName: \"kubernetes.io/projected/334663b8-eb31-4dd4-98ff-734383066552-kube-api-access-cnlsd\") pod \"managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7\" (UID: \"334663b8-eb31-4dd4-98ff-734383066552\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7"
Sep 16 18:42:08.264404 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.264403    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/334663b8-eb31-4dd4-98ff-734383066552-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7\" (UID: \"334663b8-eb31-4dd4-98ff-734383066552\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7"
Sep 16 18:42:08.266959 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.266933    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/334663b8-eb31-4dd4-98ff-734383066552-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7\" (UID: \"334663b8-eb31-4dd4-98ff-734383066552\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7"
Sep 16 18:42:08.279600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.279563    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnlsd\" (UniqueName: \"kubernetes.io/projected/334663b8-eb31-4dd4-98ff-734383066552-kube-api-access-cnlsd\") pod \"managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7\" (UID: \"334663b8-eb31-4dd4-98ff-734383066552\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7"
Sep 16 18:42:08.402764 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.402727    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7"
Sep 16 18:42:08.683129 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:08.683025    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7"]
Sep 16 18:42:08.686677 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:42:08.686648    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod334663b8_eb31_4dd4_98ff_734383066552.slice/crio-67d7c50f42991fab431b2cb875bad193200211202787588cf0abff4f576e9354 WatchSource:0}: Error finding container 67d7c50f42991fab431b2cb875bad193200211202787588cf0abff4f576e9354: Status 404 returned error can't find the container with id 67d7c50f42991fab431b2cb875bad193200211202787588cf0abff4f576e9354
Sep 16 18:42:09.169642 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:09.169610    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7" event={"ID":"334663b8-eb31-4dd4-98ff-734383066552","Type":"ContainerStarted","Data":"67d7c50f42991fab431b2cb875bad193200211202787588cf0abff4f576e9354"}
Sep 16 18:42:12.181726 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.181694    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-95f4c55d6-jw47r"]
Sep 16 18:42:12.184627 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.184599    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7" event={"ID":"334663b8-eb31-4dd4-98ff-734383066552","Type":"ContainerStarted","Data":"4c2c2b05a023e5c09305333edc37ec041330d9f5b78c5a0ace999b2eb6067b50"}
Sep 16 18:42:12.184775 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.184678    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.187166 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.187140    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:42:12.188012 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.187994    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:42:12.188130 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.188025    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:42:12.188130 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.188027    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gfxsd\""
Sep 16 18:42:12.188336 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.188321    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:42:12.188413 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.188336    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:42:12.188413 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.188348    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:42:12.188413 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.188338    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:42:12.196630 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.196598    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95f4c55d6-jw47r"]
Sep 16 18:42:12.204807 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.204754    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67c49bf5c4-9pnn7" podStartSLOduration=1.127991111 podStartE2EDuration="4.204736308s" totalImagesPullingTime="3.076745197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:12.204477466 +0000 UTC m=+128.098664736" watchObservedRunningTime="2026-09-16 18:42:12.204736308 +0000 UTC m=+128.098923576"
Sep 16 18:42:12.297327 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.297276    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d40557c-52e1-4932-af16-4c38702a2a87-console-serving-cert\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.297501 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.297338    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-service-ca\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.297501 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.297379    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-oauth-serving-cert\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.297501 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.297478    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d40557c-52e1-4932-af16-4c38702a2a87-console-oauth-config\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.297606 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.297544    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npk6k\" (UniqueName: \"kubernetes.io/projected/4d40557c-52e1-4932-af16-4c38702a2a87-kube-api-access-npk6k\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.297640 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.297623    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-console-config\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.399002 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.398952    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-console-config\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.399142 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.399014    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d40557c-52e1-4932-af16-4c38702a2a87-console-serving-cert\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.399142 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.399042    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-service-ca\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.399142 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.399061    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-oauth-serving-cert\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.399142 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.399076    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d40557c-52e1-4932-af16-4c38702a2a87-console-oauth-config\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.399142 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.399115    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npk6k\" (UniqueName: \"kubernetes.io/projected/4d40557c-52e1-4932-af16-4c38702a2a87-kube-api-access-npk6k\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.399783 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.399756    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-console-config\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.399881 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.399757    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-oauth-serving-cert\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.399881 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.399835    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-service-ca\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.401597 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.401569    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d40557c-52e1-4932-af16-4c38702a2a87-console-oauth-config\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.401700 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.401653    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d40557c-52e1-4932-af16-4c38702a2a87-console-serving-cert\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.411816 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.411783    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npk6k\" (UniqueName: \"kubernetes.io/projected/4d40557c-52e1-4932-af16-4c38702a2a87-kube-api-access-npk6k\") pod \"console-95f4c55d6-jw47r\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") " pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.494178 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.494064    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:12.769515 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.769431    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95f4c55d6-jw47r"]
Sep 16 18:42:12.772459 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:42:12.772431    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d40557c_52e1_4932_af16_4c38702a2a87.slice/crio-c3507023d07ddc82f848a76251bd16155a290dc4462d17022cf3fa3f0f398903 WatchSource:0}: Error finding container c3507023d07ddc82f848a76251bd16155a290dc4462d17022cf3fa3f0f398903: Status 404 returned error can't find the container with id c3507023d07ddc82f848a76251bd16155a290dc4462d17022cf3fa3f0f398903
Sep 16 18:42:12.956431 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.956392    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pmwbl"]
Sep 16 18:42:12.960574 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.960536    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pmwbl"
Sep 16 18:42:12.963551 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.963521    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:42:12.963683 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.963661    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rmsms\""
Sep 16 18:42:12.985408 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:12.985374    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pmwbl"]
Sep 16 18:42:13.003882 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:13.003839    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/745a1716-da85-4e62-a4d4-bb680e50cc86-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pmwbl\" (UID: \"745a1716-da85-4e62-a4d4-bb680e50cc86\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pmwbl"
Sep 16 18:42:13.104379 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:13.104274    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/745a1716-da85-4e62-a4d4-bb680e50cc86-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pmwbl\" (UID: \"745a1716-da85-4e62-a4d4-bb680e50cc86\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pmwbl"
Sep 16 18:42:13.104525 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:42:13.104390    2552 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:42:13.104525 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:42:13.104446    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/745a1716-da85-4e62-a4d4-bb680e50cc86-tls-certificates podName:745a1716-da85-4e62-a4d4-bb680e50cc86 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:13.604431782 +0000 UTC m=+129.498619029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/745a1716-da85-4e62-a4d4-bb680e50cc86-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-pmwbl" (UID: "745a1716-da85-4e62-a4d4-bb680e50cc86") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:42:13.188172 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:13.188138    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95f4c55d6-jw47r" event={"ID":"4d40557c-52e1-4932-af16-4c38702a2a87","Type":"ContainerStarted","Data":"c3507023d07ddc82f848a76251bd16155a290dc4462d17022cf3fa3f0f398903"}
Sep 16 18:42:13.608159 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:13.608112    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/745a1716-da85-4e62-a4d4-bb680e50cc86-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pmwbl\" (UID: \"745a1716-da85-4e62-a4d4-bb680e50cc86\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pmwbl"
Sep 16 18:42:13.610979 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:13.610949    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/745a1716-da85-4e62-a4d4-bb680e50cc86-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pmwbl\" (UID: \"745a1716-da85-4e62-a4d4-bb680e50cc86\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pmwbl"
Sep 16 18:42:13.870438 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:13.870335    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pmwbl"
Sep 16 18:42:14.157681 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:14.157645    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pmwbl"]
Sep 16 18:42:14.161795 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:42:14.161768    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745a1716_da85_4e62_a4d4_bb680e50cc86.slice/crio-a11ab1821ca585e27e3020ec84c01736ee8777643a9f4219304c0db06f930676 WatchSource:0}: Error finding container a11ab1821ca585e27e3020ec84c01736ee8777643a9f4219304c0db06f930676: Status 404 returned error can't find the container with id a11ab1821ca585e27e3020ec84c01736ee8777643a9f4219304c0db06f930676
Sep 16 18:42:14.193599 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:14.193553    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pmwbl" event={"ID":"745a1716-da85-4e62-a4d4-bb680e50cc86","Type":"ContainerStarted","Data":"a11ab1821ca585e27e3020ec84c01736ee8777643a9f4219304c0db06f930676"}
Sep 16 18:42:14.315385 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:14.315339    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs\") pod \"network-metrics-daemon-zkgpv\" (UID: \"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c\") " pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:42:14.317803 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:14.317771    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5f1c97-1813-4d89-8af7-ce7d0eebc10c-metrics-certs\") pod \"network-metrics-daemon-zkgpv\" (UID: \"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c\") " pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:42:14.449056 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:14.448973    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-md6w5\""
Sep 16 18:42:14.457433 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:14.457400    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkgpv"
Sep 16 18:42:14.743346 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:14.743262    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zkgpv"]
Sep 16 18:42:16.036809 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:42:16.036765    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5f1c97_1813_4d89_8af7_ce7d0eebc10c.slice/crio-5e4c3a78f30e79bf5f6c44ec7158073592c69b44f0671d9ccc415bcbc1c18bba WatchSource:0}: Error finding container 5e4c3a78f30e79bf5f6c44ec7158073592c69b44f0671d9ccc415bcbc1c18bba: Status 404 returned error can't find the container with id 5e4c3a78f30e79bf5f6c44ec7158073592c69b44f0671d9ccc415bcbc1c18bba
Sep 16 18:42:16.203557 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:16.203499    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zkgpv" event={"ID":"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c","Type":"ContainerStarted","Data":"5e4c3a78f30e79bf5f6c44ec7158073592c69b44f0671d9ccc415bcbc1c18bba"}
Sep 16 18:42:16.205141 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:16.205109    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pmwbl" event={"ID":"745a1716-da85-4e62-a4d4-bb680e50cc86","Type":"ContainerStarted","Data":"8008a731362fae76a9d7d0a7e279449b73d99d7bff7261e04a3f8c5427524328"}
Sep 16 18:42:16.205297 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:16.205241    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pmwbl"
Sep 16 18:42:16.206869 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:16.206837    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95f4c55d6-jw47r" event={"ID":"4d40557c-52e1-4932-af16-4c38702a2a87","Type":"ContainerStarted","Data":"de6bc95a2283ab4a909eb45e7414b5a3a70978fcbf974e307f32250376a843ed"}
Sep 16 18:42:16.210353 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:16.210327    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pmwbl"
Sep 16 18:42:16.223108 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:16.223040    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pmwbl" podStartSLOduration=2.298875655 podStartE2EDuration="4.223022161s" totalImagesPullingTime="1.924146506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:16.22204736 +0000 UTC m=+132.116234629" watchObservedRunningTime="2026-09-16 18:42:16.223022161 +0000 UTC m=+132.117209430"
Sep 16 18:42:16.261756 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:16.261689    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-95f4c55d6-jw47r" podStartSLOduration=0.946269047 podStartE2EDuration="4.26166678s" totalImagesPullingTime="3.315397733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:16.260806633 +0000 UTC m=+132.154993903" watchObservedRunningTime="2026-09-16 18:42:16.26166678 +0000 UTC m=+132.155854050"
Sep 16 18:42:17.068753 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.068713    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"]
Sep 16 18:42:17.072612 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.072579    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"
Sep 16 18:42:17.078555 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.078340    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:42:17.078555 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.078340    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:42:17.078555 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.078539    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5xgxw\""
Sep 16 18:42:17.078857 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.078547    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:42:17.083040 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.082975    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:42:17.085024 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.084998    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"]
Sep 16 18:42:17.087082 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.087058    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:42:17.139797 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.139770    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/842b872c-6600-4b43-9ca0-8ae13cd550d8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-rxzfb\" (UID: \"842b872c-6600-4b43-9ca0-8ae13cd550d8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"
Sep 16 18:42:17.139878 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.139826    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/842b872c-6600-4b43-9ca0-8ae13cd550d8-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-rxzfb\" (UID: \"842b872c-6600-4b43-9ca0-8ae13cd550d8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"
Sep 16 18:42:17.139957 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.139943    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jprxk\" (UniqueName: \"kubernetes.io/projected/842b872c-6600-4b43-9ca0-8ae13cd550d8-kube-api-access-jprxk\") pod \"prometheus-operator-68bcd7f987-rxzfb\" (UID: \"842b872c-6600-4b43-9ca0-8ae13cd550d8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"
Sep 16 18:42:17.140013 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.139978    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/842b872c-6600-4b43-9ca0-8ae13cd550d8-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-rxzfb\" (UID: \"842b872c-6600-4b43-9ca0-8ae13cd550d8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"
Sep 16 18:42:17.241253 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.241220    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jprxk\" (UniqueName: \"kubernetes.io/projected/842b872c-6600-4b43-9ca0-8ae13cd550d8-kube-api-access-jprxk\") pod \"prometheus-operator-68bcd7f987-rxzfb\" (UID: \"842b872c-6600-4b43-9ca0-8ae13cd550d8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"
Sep 16 18:42:17.241397 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.241259    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/842b872c-6600-4b43-9ca0-8ae13cd550d8-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-rxzfb\" (UID: \"842b872c-6600-4b43-9ca0-8ae13cd550d8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"
Sep 16 18:42:17.241397 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.241307    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/842b872c-6600-4b43-9ca0-8ae13cd550d8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-rxzfb\" (UID: \"842b872c-6600-4b43-9ca0-8ae13cd550d8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"
Sep 16 18:42:17.241397 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.241345    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/842b872c-6600-4b43-9ca0-8ae13cd550d8-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-rxzfb\" (UID: \"842b872c-6600-4b43-9ca0-8ae13cd550d8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"
Sep 16 18:42:17.241968 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.241943    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/842b872c-6600-4b43-9ca0-8ae13cd550d8-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-rxzfb\" (UID: \"842b872c-6600-4b43-9ca0-8ae13cd550d8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"
Sep 16 18:42:17.243952 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.243920    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/842b872c-6600-4b43-9ca0-8ae13cd550d8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-rxzfb\" (UID: \"842b872c-6600-4b43-9ca0-8ae13cd550d8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"
Sep 16 18:42:17.244303 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.244280    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/842b872c-6600-4b43-9ca0-8ae13cd550d8-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-rxzfb\" (UID: \"842b872c-6600-4b43-9ca0-8ae13cd550d8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"
Sep 16 18:42:17.256606 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.256571    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jprxk\" (UniqueName: \"kubernetes.io/projected/842b872c-6600-4b43-9ca0-8ae13cd550d8-kube-api-access-jprxk\") pod \"prometheus-operator-68bcd7f987-rxzfb\" (UID: \"842b872c-6600-4b43-9ca0-8ae13cd550d8\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"
Sep 16 18:42:17.383944 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.383901    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"
Sep 16 18:42:17.661056 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:42:17.661019    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842b872c_6600_4b43_9ca0_8ae13cd550d8.slice/crio-3ad5e8142b526a29693c9ab9bab58b514869515806524e39a04ec663699f6c81 WatchSource:0}: Error finding container 3ad5e8142b526a29693c9ab9bab58b514869515806524e39a04ec663699f6c81: Status 404 returned error can't find the container with id 3ad5e8142b526a29693c9ab9bab58b514869515806524e39a04ec663699f6c81
Sep 16 18:42:17.667556 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:17.665220    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb"]
Sep 16 18:42:18.215855 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:18.215802    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb" event={"ID":"842b872c-6600-4b43-9ca0-8ae13cd550d8","Type":"ContainerStarted","Data":"3ad5e8142b526a29693c9ab9bab58b514869515806524e39a04ec663699f6c81"}
Sep 16 18:42:18.218111 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:18.218052    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zkgpv" event={"ID":"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c","Type":"ContainerStarted","Data":"6b3ee03261b4e78697db78ee59844f25da551206dba7eea0b8a4f4b62e453537"}
Sep 16 18:42:18.218274 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:18.218142    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zkgpv" event={"ID":"0f5f1c97-1813-4d89-8af7-ce7d0eebc10c","Type":"ContainerStarted","Data":"73a666f269f0303d1f2458b50ace709b9ec6ad96f3b141d71131283dc975d3d9"}
Sep 16 18:42:18.235448 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:18.235380    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zkgpv" podStartSLOduration=132.178541718 podStartE2EDuration="2m13.235357948s" totalImagesPullingTime="1.05681623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:18.235288842 +0000 UTC m=+134.129476116" watchObservedRunningTime="2026-09-16 18:42:18.235357948 +0000 UTC m=+134.129545218"
Sep 16 18:42:19.185217 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.185176    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57c58dbd54-sc69h"]
Sep 16 18:42:19.188354 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.188330    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.195746 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.195717    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:42:19.208132 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.208082    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57c58dbd54-sc69h"]
Sep 16 18:42:19.223658 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.223624    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb" event={"ID":"842b872c-6600-4b43-9ca0-8ae13cd550d8","Type":"ContainerStarted","Data":"827abfe76245d6b4b41806cf1f085b2b06e57f33fcf23c3e0a32794f8a71a1c6"}
Sep 16 18:42:19.224043 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.223686    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb" event={"ID":"842b872c-6600-4b43-9ca0-8ae13cd550d8","Type":"ContainerStarted","Data":"7a0151abf94ff50b8e57a77b34d897729880857733054b339f026a33a306d264"}
Sep 16 18:42:19.261677 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.261635    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-oauth-serving-cert\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.261859 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.261700    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-trusted-ca-bundle\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.261859 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.261722    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-console-config\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.261859 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.261761    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f92f32e-9616-474d-809d-b90ae021a853-console-serving-cert\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.261859 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.261792    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f92f32e-9616-474d-809d-b90ae021a853-console-oauth-config\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.261859 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.261811    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhsb7\" (UniqueName: \"kubernetes.io/projected/1f92f32e-9616-474d-809d-b90ae021a853-kube-api-access-dhsb7\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.261859 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.261830    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-service-ca\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.265628 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.265577    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rxzfb" podStartSLOduration=1.08341463 podStartE2EDuration="2.265559245s" totalImagesPullingTime="1.182144615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:19.265251586 +0000 UTC m=+135.159438857" watchObservedRunningTime="2026-09-16 18:42:19.265559245 +0000 UTC m=+135.159746513"
Sep 16 18:42:19.362660 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.362618    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f92f32e-9616-474d-809d-b90ae021a853-console-oauth-config\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.362928 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.362770    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhsb7\" (UniqueName: \"kubernetes.io/projected/1f92f32e-9616-474d-809d-b90ae021a853-kube-api-access-dhsb7\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.362928 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.362812    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-service-ca\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.362928 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.362859    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-oauth-serving-cert\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.363134 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.362965    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-trusted-ca-bundle\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.363134 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.363006    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-console-config\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.363134 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.363038    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f92f32e-9616-474d-809d-b90ae021a853-console-serving-cert\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.363729 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.363698    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-service-ca\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.363729 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.363722    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-oauth-serving-cert\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.363896 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.363747    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-console-config\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.364394 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.364366    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-trusted-ca-bundle\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.365369 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.365344    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f92f32e-9616-474d-809d-b90ae021a853-console-oauth-config\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.365586 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.365569    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f92f32e-9616-474d-809d-b90ae021a853-console-serving-cert\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.372386 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.372358    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhsb7\" (UniqueName: \"kubernetes.io/projected/1f92f32e-9616-474d-809d-b90ae021a853-kube-api-access-dhsb7\") pod \"console-57c58dbd54-sc69h\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") " pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.497324 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.497236    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:19.772228 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:19.772130    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57c58dbd54-sc69h"]
Sep 16 18:42:19.776187 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:42:19.776150    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f92f32e_9616_474d_809d_b90ae021a853.slice/crio-4b78e58376171f4d89ab3a8abd3cc805c04bff3fec8785a0ec4cb79c4dbe0bba WatchSource:0}: Error finding container 4b78e58376171f4d89ab3a8abd3cc805c04bff3fec8785a0ec4cb79c4dbe0bba: Status 404 returned error can't find the container with id 4b78e58376171f4d89ab3a8abd3cc805c04bff3fec8785a0ec4cb79c4dbe0bba
Sep 16 18:42:20.228987 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:20.228947    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c58dbd54-sc69h" event={"ID":"1f92f32e-9616-474d-809d-b90ae021a853","Type":"ContainerStarted","Data":"3863fc5ee530a01090dfc0ce745c128d40d5e78906a08e2c3f3230069ac086ff"}
Sep 16 18:42:20.229372 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:20.229008    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c58dbd54-sc69h" event={"ID":"1f92f32e-9616-474d-809d-b90ae021a853","Type":"ContainerStarted","Data":"4b78e58376171f4d89ab3a8abd3cc805c04bff3fec8785a0ec4cb79c4dbe0bba"}
Sep 16 18:42:20.247233 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:20.247178    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57c58dbd54-sc69h" podStartSLOduration=1.247159643 podStartE2EDuration="1.247159643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:20.246260641 +0000 UTC m=+136.140447910" watchObservedRunningTime="2026-09-16 18:42:20.247159643 +0000 UTC m=+136.141346912"
Sep 16 18:42:21.442309 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.442268    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"]
Sep 16 18:42:21.445759 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.445737    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"
Sep 16 18:42:21.448440 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.448409    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 18:42:21.448440 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.448435    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-pbzw5\""
Sep 16 18:42:21.448630 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.448435    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:42:21.458478 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.458449    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"]
Sep 16 18:42:21.513993 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.513954    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qj79k"]
Sep 16 18:42:21.517215 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.517191    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.519695 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.519667    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:42:21.519695 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.519677    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:42:21.519695 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.519689    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:42:21.519929 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.519726    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7vh8q\""
Sep 16 18:42:21.581706 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.581668    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e47c5d54-4b22-4722-bb8c-b1ef622ac42f-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-r5c45\" (UID: \"e47c5d54-4b22-4722-bb8c-b1ef622ac42f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"
Sep 16 18:42:21.581706 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.581706    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e47c5d54-4b22-4722-bb8c-b1ef622ac42f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-r5c45\" (UID: \"e47c5d54-4b22-4722-bb8c-b1ef622ac42f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"
Sep 16 18:42:21.581911 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.581736    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-node-exporter-textfile\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.581911 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.581845    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-sys\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.581911 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.581877    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-node-exporter-accelerators-collector-config\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.581911 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.581908    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-node-exporter-tls\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.582149 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.581925    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-node-exporter-wtmp\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.582149 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.581986    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-root\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.582149 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.582028    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwgj4\" (UniqueName: \"kubernetes.io/projected/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-kube-api-access-hwgj4\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.582149 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.582061    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.582149 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.582135    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e47c5d54-4b22-4722-bb8c-b1ef622ac42f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-r5c45\" (UID: \"e47c5d54-4b22-4722-bb8c-b1ef622ac42f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"
Sep 16 18:42:21.582359 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.582166    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djqc6\" (UniqueName: \"kubernetes.io/projected/e47c5d54-4b22-4722-bb8c-b1ef622ac42f-kube-api-access-djqc6\") pod \"openshift-state-metrics-67dcf8c979-r5c45\" (UID: \"e47c5d54-4b22-4722-bb8c-b1ef622ac42f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"
Sep 16 18:42:21.582359 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.582196    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-metrics-client-ca\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.682945 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.682902    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e47c5d54-4b22-4722-bb8c-b1ef622ac42f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-r5c45\" (UID: \"e47c5d54-4b22-4722-bb8c-b1ef622ac42f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"
Sep 16 18:42:21.682945 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.682946    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-node-exporter-textfile\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.683209 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683011    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-sys\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.683209 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683038    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-node-exporter-accelerators-collector-config\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.683209 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:42:21.683098    2552 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 16 18:42:21.683209 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683147    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-sys\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.683209 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683105    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-node-exporter-tls\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.683209 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:42:21.683190    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e47c5d54-4b22-4722-bb8c-b1ef622ac42f-openshift-state-metrics-tls podName:e47c5d54-4b22-4722-bb8c-b1ef622ac42f nodeName:}" failed. No retries permitted until 2026-09-16 18:42:22.183169294 +0000 UTC m=+138.077356554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e47c5d54-4b22-4722-bb8c-b1ef622ac42f-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-r5c45" (UID: "e47c5d54-4b22-4722-bb8c-b1ef622ac42f") : secret "openshift-state-metrics-tls" not found
Sep 16 18:42:21.683505 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683376    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-node-exporter-textfile\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.683505 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683382    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-node-exporter-wtmp\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.683505 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683439    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-root\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.683505 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683464    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwgj4\" (UniqueName: \"kubernetes.io/projected/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-kube-api-access-hwgj4\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.683703 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683505    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-root\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.683703 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683526    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-node-exporter-wtmp\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.683703 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683634    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.683703 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683672    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e47c5d54-4b22-4722-bb8c-b1ef622ac42f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-r5c45\" (UID: \"e47c5d54-4b22-4722-bb8c-b1ef622ac42f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"
Sep 16 18:42:21.683883 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683742    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-node-exporter-accelerators-collector-config\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.683883 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683788    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djqc6\" (UniqueName: \"kubernetes.io/projected/e47c5d54-4b22-4722-bb8c-b1ef622ac42f-kube-api-access-djqc6\") pod \"openshift-state-metrics-67dcf8c979-r5c45\" (UID: \"e47c5d54-4b22-4722-bb8c-b1ef622ac42f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"
Sep 16 18:42:21.683883 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683857    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-metrics-client-ca\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.684035 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.683932    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e47c5d54-4b22-4722-bb8c-b1ef622ac42f-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-r5c45\" (UID: \"e47c5d54-4b22-4722-bb8c-b1ef622ac42f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"
Sep 16 18:42:21.684522 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.684497    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-metrics-client-ca\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.684772 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.684746    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e47c5d54-4b22-4722-bb8c-b1ef622ac42f-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-r5c45\" (UID: \"e47c5d54-4b22-4722-bb8c-b1ef622ac42f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"
Sep 16 18:42:21.685886 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.685859    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-node-exporter-tls\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.686031 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.686012    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.686117 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.686054    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e47c5d54-4b22-4722-bb8c-b1ef622ac42f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-r5c45\" (UID: \"e47c5d54-4b22-4722-bb8c-b1ef622ac42f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"
Sep 16 18:42:21.696179 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.696103    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwgj4\" (UniqueName: \"kubernetes.io/projected/032307cc-4c2d-4ff7-9be6-4b9baadd54fb-kube-api-access-hwgj4\") pod \"node-exporter-qj79k\" (UID: \"032307cc-4c2d-4ff7-9be6-4b9baadd54fb\") " pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.696767 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.696746    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djqc6\" (UniqueName: \"kubernetes.io/projected/e47c5d54-4b22-4722-bb8c-b1ef622ac42f-kube-api-access-djqc6\") pod \"openshift-state-metrics-67dcf8c979-r5c45\" (UID: \"e47c5d54-4b22-4722-bb8c-b1ef622ac42f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"
Sep 16 18:42:21.826438 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:21.826395    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qj79k"
Sep 16 18:42:21.838153 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:42:21.838110    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032307cc_4c2d_4ff7_9be6_4b9baadd54fb.slice/crio-2c29b2da319c8de904298a5a47bd0ffc98f34e364f70919778ffa7c8ac82f26d WatchSource:0}: Error finding container 2c29b2da319c8de904298a5a47bd0ffc98f34e364f70919778ffa7c8ac82f26d: Status 404 returned error can't find the container with id 2c29b2da319c8de904298a5a47bd0ffc98f34e364f70919778ffa7c8ac82f26d
Sep 16 18:42:22.188876 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:22.188822    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e47c5d54-4b22-4722-bb8c-b1ef622ac42f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-r5c45\" (UID: \"e47c5d54-4b22-4722-bb8c-b1ef622ac42f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"
Sep 16 18:42:22.191816 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:22.191782    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e47c5d54-4b22-4722-bb8c-b1ef622ac42f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-r5c45\" (UID: \"e47c5d54-4b22-4722-bb8c-b1ef622ac42f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"
Sep 16 18:42:22.238752 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:22.238707    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qj79k" event={"ID":"032307cc-4c2d-4ff7-9be6-4b9baadd54fb","Type":"ContainerStarted","Data":"2c29b2da319c8de904298a5a47bd0ffc98f34e364f70919778ffa7c8ac82f26d"}
Sep 16 18:42:22.355699 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:22.355658    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"
Sep 16 18:42:22.494820 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:22.494725    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:22.494820 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:22.494805    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:22.500478 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:22.500423    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:22.648573 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:22.648535    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45"]
Sep 16 18:42:22.652754 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:42:22.652727    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47c5d54_4b22_4722_bb8c_b1ef622ac42f.slice/crio-a51f6d36e97c3fe5d6e25c63b6a2aeb6b1aec2954e7971431e633f166493bb70 WatchSource:0}: Error finding container a51f6d36e97c3fe5d6e25c63b6a2aeb6b1aec2954e7971431e633f166493bb70: Status 404 returned error can't find the container with id a51f6d36e97c3fe5d6e25c63b6a2aeb6b1aec2954e7971431e633f166493bb70
Sep 16 18:42:23.245017 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:23.244974    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45" event={"ID":"e47c5d54-4b22-4722-bb8c-b1ef622ac42f","Type":"ContainerStarted","Data":"f041f0b22f39bcdedaec53afcf742910149eb92a0f756ed583876ffd5bfbc082"}
Sep 16 18:42:23.245244 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:23.245043    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45" event={"ID":"e47c5d54-4b22-4722-bb8c-b1ef622ac42f","Type":"ContainerStarted","Data":"3f4cfcf2c0b8efe8d3dc9323e217d00d1e048de78371e923376bbf4d2f6e65cd"}
Sep 16 18:42:23.245244 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:23.245067    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45" event={"ID":"e47c5d54-4b22-4722-bb8c-b1ef622ac42f","Type":"ContainerStarted","Data":"a51f6d36e97c3fe5d6e25c63b6a2aeb6b1aec2954e7971431e633f166493bb70"}
Sep 16 18:42:23.247010 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:23.246962    2552 generic.go:356] "Generic (PLEG): container finished" podID="032307cc-4c2d-4ff7-9be6-4b9baadd54fb" containerID="816d80ac59b4193394b340f11add277680ad8d898b5ae20ec86391a3196403b8" exitCode=0
Sep 16 18:42:23.247156 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:23.247052    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qj79k" event={"ID":"032307cc-4c2d-4ff7-9be6-4b9baadd54fb","Type":"ContainerDied","Data":"816d80ac59b4193394b340f11add277680ad8d898b5ae20ec86391a3196403b8"}
Sep 16 18:42:23.255818 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:23.252554    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:42:24.253012 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:24.252976    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45" event={"ID":"e47c5d54-4b22-4722-bb8c-b1ef622ac42f","Type":"ContainerStarted","Data":"b7232ddeb1837ef955417c0bdf97ca27a8d04c685122a5e7b3d14360fe8812e0"}
Sep 16 18:42:24.255443 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:24.255412    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qj79k" event={"ID":"032307cc-4c2d-4ff7-9be6-4b9baadd54fb","Type":"ContainerStarted","Data":"c2e338bb571f3b2bee787d5fe5944f25acb947c90bdd4d40ffb1711df8a75858"}
Sep 16 18:42:24.255589 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:24.255474    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qj79k" event={"ID":"032307cc-4c2d-4ff7-9be6-4b9baadd54fb","Type":"ContainerStarted","Data":"e2d3f5cec1426396e27be17e0cf9d553aa2aa9a48316e0cdcf913bfa01999f9a"}
Sep 16 18:42:24.271870 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:24.271813    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-r5c45" podStartSLOduration=2.432949509 podStartE2EDuration="3.271794989s" totalImagesPullingTime="838.84548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:24.270362347 +0000 UTC m=+140.164549611" watchObservedRunningTime="2026-09-16 18:42:24.271794989 +0000 UTC m=+140.165982258"
Sep 16 18:42:24.297435 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:24.297371    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qj79k" podStartSLOduration=2.378834172 podStartE2EDuration="3.297352202s" totalImagesPullingTime="843.338482ms" totalInitContainerRuntime="75.179548ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:24.296152157 +0000 UTC m=+140.190339430" watchObservedRunningTime="2026-09-16 18:42:24.297352202 +0000 UTC m=+140.191539471"
Sep 16 18:42:25.780226 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.780187    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"]
Sep 16 18:42:25.783451 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.783429    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:25.786654 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.786617    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:42:25.787615 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.787586    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:42:25.787735 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.787626    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:42:25.787735 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.787691    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:42:25.788360 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.788334    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-aphjblavsj6hq\""
Sep 16 18:42:25.788577 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.788560    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-f5z2g\""
Sep 16 18:42:25.799110 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.797414    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"]
Sep 16 18:42:25.925066 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.925024    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bbe0642e-3461-40d3-bee5-6be6545fc3d1-metrics-server-audit-profiles\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:25.925066 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.925066    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx9ks\" (UniqueName: \"kubernetes.io/projected/bbe0642e-3461-40d3-bee5-6be6545fc3d1-kube-api-access-tx9ks\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:25.925312 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.925127    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bbe0642e-3461-40d3-bee5-6be6545fc3d1-audit-log\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:25.925312 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.925153    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe0642e-3461-40d3-bee5-6be6545fc3d1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:25.925312 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.925199    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bbe0642e-3461-40d3-bee5-6be6545fc3d1-secret-metrics-server-client-certs\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:25.925312 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.925216    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bbe0642e-3461-40d3-bee5-6be6545fc3d1-client-ca-bundle\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:25.925439 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:25.925321    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bbe0642e-3461-40d3-bee5-6be6545fc3d1-secret-metrics-server-tls\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.026694 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.026649    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe0642e-3461-40d3-bee5-6be6545fc3d1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.026873 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.026718    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bbe0642e-3461-40d3-bee5-6be6545fc3d1-secret-metrics-server-client-certs\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.026873 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.026739    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bbe0642e-3461-40d3-bee5-6be6545fc3d1-client-ca-bundle\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.026873 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.026767    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bbe0642e-3461-40d3-bee5-6be6545fc3d1-secret-metrics-server-tls\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.027022 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.026941    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bbe0642e-3461-40d3-bee5-6be6545fc3d1-metrics-server-audit-profiles\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.027022 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.026992    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx9ks\" (UniqueName: \"kubernetes.io/projected/bbe0642e-3461-40d3-bee5-6be6545fc3d1-kube-api-access-tx9ks\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.027150 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.027046    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bbe0642e-3461-40d3-bee5-6be6545fc3d1-audit-log\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.027488 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.027455    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bbe0642e-3461-40d3-bee5-6be6545fc3d1-audit-log\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.027623 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.027520    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe0642e-3461-40d3-bee5-6be6545fc3d1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.028027 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.028000    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bbe0642e-3461-40d3-bee5-6be6545fc3d1-metrics-server-audit-profiles\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.029401 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.029367    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bbe0642e-3461-40d3-bee5-6be6545fc3d1-secret-metrics-server-client-certs\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.029563 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.029541    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bbe0642e-3461-40d3-bee5-6be6545fc3d1-secret-metrics-server-tls\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.029612 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.029575    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bbe0642e-3461-40d3-bee5-6be6545fc3d1-client-ca-bundle\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.036762 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.036703    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx9ks\" (UniqueName: \"kubernetes.io/projected/bbe0642e-3461-40d3-bee5-6be6545fc3d1-kube-api-access-tx9ks\") pod \"metrics-server-54b8f9ff88-lkgg8\" (UID: \"bbe0642e-3461-40d3-bee5-6be6545fc3d1\") " pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.093742 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.093703    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:26.196714 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.196673    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp"]
Sep 16 18:42:26.199983 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.199957    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp"
Sep 16 18:42:26.202710 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.202678    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 18:42:26.202710 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.202695    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5nx7t\""
Sep 16 18:42:26.209688 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.209658    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp"]
Sep 16 18:42:26.329328 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.329241    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d67daef5-60ea-49a2-8a86-a6bf94dcebbc-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-kbwnp\" (UID: \"d67daef5-60ea-49a2-8a86-a6bf94dcebbc\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp"
Sep 16 18:42:26.329328 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.329288    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkf5k\" (UniqueName: \"kubernetes.io/projected/d67daef5-60ea-49a2-8a86-a6bf94dcebbc-kube-api-access-wkf5k\") pod \"monitoring-plugin-5d85778b9d-kbwnp\" (UID: \"d67daef5-60ea-49a2-8a86-a6bf94dcebbc\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp"
Sep 16 18:42:26.375567 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.375539    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"]
Sep 16 18:42:26.378162 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:42:26.378128    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe0642e_3461_40d3_bee5_6be6545fc3d1.slice/crio-6a571f57ae541228e95d9bf2b7b04cab25727d6c2c7a439b4e2c60809ada9a63 WatchSource:0}: Error finding container 6a571f57ae541228e95d9bf2b7b04cab25727d6c2c7a439b4e2c60809ada9a63: Status 404 returned error can't find the container with id 6a571f57ae541228e95d9bf2b7b04cab25727d6c2c7a439b4e2c60809ada9a63
Sep 16 18:42:26.429911 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.429870    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkf5k\" (UniqueName: \"kubernetes.io/projected/d67daef5-60ea-49a2-8a86-a6bf94dcebbc-kube-api-access-wkf5k\") pod \"monitoring-plugin-5d85778b9d-kbwnp\" (UID: \"d67daef5-60ea-49a2-8a86-a6bf94dcebbc\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp"
Sep 16 18:42:26.430122 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.429985    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d67daef5-60ea-49a2-8a86-a6bf94dcebbc-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-kbwnp\" (UID: \"d67daef5-60ea-49a2-8a86-a6bf94dcebbc\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp"
Sep 16 18:42:26.430122 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:42:26.430110    2552 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 18:42:26.430218 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:42:26.430174    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d67daef5-60ea-49a2-8a86-a6bf94dcebbc-monitoring-plugin-cert podName:d67daef5-60ea-49a2-8a86-a6bf94dcebbc nodeName:}" failed. No retries permitted until 2026-09-16 18:42:26.930160094 +0000 UTC m=+142.824347340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d67daef5-60ea-49a2-8a86-a6bf94dcebbc-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-kbwnp" (UID: "d67daef5-60ea-49a2-8a86-a6bf94dcebbc") : secret "monitoring-plugin-cert" not found
Sep 16 18:42:26.439355 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.439331    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkf5k\" (UniqueName: \"kubernetes.io/projected/d67daef5-60ea-49a2-8a86-a6bf94dcebbc-kube-api-access-wkf5k\") pod \"monitoring-plugin-5d85778b9d-kbwnp\" (UID: \"d67daef5-60ea-49a2-8a86-a6bf94dcebbc\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp"
Sep 16 18:42:26.723934 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.723888    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-75cc585756-x9qwr"]
Sep 16 18:42:26.727539 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.727509    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.729821 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.729796    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-dc47m\""
Sep 16 18:42:26.729821 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.729809    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 18:42:26.730046 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.729836    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 18:42:26.730203 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.730175    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 18:42:26.730353 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.730196    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 18:42:26.730353 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.730219    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 18:42:26.735544 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.735514    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 18:42:26.741355 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.741320    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75cc585756-x9qwr"]
Sep 16 18:42:26.833807 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.833768    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f7055b-0255-4a0f-a1f6-971cae2af884-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.834270 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.833839    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/36f7055b-0255-4a0f-a1f6-971cae2af884-federate-client-tls\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.834270 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.833869    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shzgl\" (UniqueName: \"kubernetes.io/projected/36f7055b-0255-4a0f-a1f6-971cae2af884-kube-api-access-shzgl\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.834270 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.833903    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f7055b-0255-4a0f-a1f6-971cae2af884-serving-certs-ca-bundle\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.834270 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.834000    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/36f7055b-0255-4a0f-a1f6-971cae2af884-secret-telemeter-client\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.834270 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.834048    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36f7055b-0255-4a0f-a1f6-971cae2af884-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.834270 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.834124    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/36f7055b-0255-4a0f-a1f6-971cae2af884-telemeter-client-tls\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.834270 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.834141    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36f7055b-0255-4a0f-a1f6-971cae2af884-metrics-client-ca\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.935642 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.935582    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shzgl\" (UniqueName: \"kubernetes.io/projected/36f7055b-0255-4a0f-a1f6-971cae2af884-kube-api-access-shzgl\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.935881 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.935671    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f7055b-0255-4a0f-a1f6-971cae2af884-serving-certs-ca-bundle\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.935881 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.935735    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/36f7055b-0255-4a0f-a1f6-971cae2af884-secret-telemeter-client\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.935881 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.935781    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36f7055b-0255-4a0f-a1f6-971cae2af884-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.935881 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.935821    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/36f7055b-0255-4a0f-a1f6-971cae2af884-telemeter-client-tls\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.935881 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.935848    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36f7055b-0255-4a0f-a1f6-971cae2af884-metrics-client-ca\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.936197 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.935885    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f7055b-0255-4a0f-a1f6-971cae2af884-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.936197 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.935917    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d67daef5-60ea-49a2-8a86-a6bf94dcebbc-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-kbwnp\" (UID: \"d67daef5-60ea-49a2-8a86-a6bf94dcebbc\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp"
Sep 16 18:42:26.936197 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.935966    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/36f7055b-0255-4a0f-a1f6-971cae2af884-federate-client-tls\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.936603 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.936570    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f7055b-0255-4a0f-a1f6-971cae2af884-serving-certs-ca-bundle\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.937032 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.937003    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f7055b-0255-4a0f-a1f6-971cae2af884-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.938037 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.938009    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36f7055b-0255-4a0f-a1f6-971cae2af884-metrics-client-ca\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.938838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.938800    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d67daef5-60ea-49a2-8a86-a6bf94dcebbc-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-kbwnp\" (UID: \"d67daef5-60ea-49a2-8a86-a6bf94dcebbc\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp"
Sep 16 18:42:26.939018 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.938994    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/36f7055b-0255-4a0f-a1f6-971cae2af884-federate-client-tls\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.939157 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.939037    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36f7055b-0255-4a0f-a1f6-971cae2af884-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.939277 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.939260    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/36f7055b-0255-4a0f-a1f6-971cae2af884-secret-telemeter-client\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.939407 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.939381    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/36f7055b-0255-4a0f-a1f6-971cae2af884-telemeter-client-tls\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:26.945468 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:26.945433    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shzgl\" (UniqueName: \"kubernetes.io/projected/36f7055b-0255-4a0f-a1f6-971cae2af884-kube-api-access-shzgl\") pod \"telemeter-client-75cc585756-x9qwr\" (UID: \"36f7055b-0255-4a0f-a1f6-971cae2af884\") " pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:27.038494 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.038415    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr"
Sep 16 18:42:27.110852 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.110727    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp"
Sep 16 18:42:27.210500 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.210453    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57c58dbd54-sc69h"]
Sep 16 18:42:27.247040 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.247006    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55cff56f6d-5twn2"]
Sep 16 18:42:27.250605 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.250576    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.265476 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.265442    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55cff56f6d-5twn2"]
Sep 16 18:42:27.269370 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.269329    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8" event={"ID":"bbe0642e-3461-40d3-bee5-6be6545fc3d1","Type":"ContainerStarted","Data":"6a571f57ae541228e95d9bf2b7b04cab25727d6c2c7a439b4e2c60809ada9a63"}
Sep 16 18:42:27.330490 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.330436    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75cc585756-x9qwr"]
Sep 16 18:42:27.333283 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:42:27.333251    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f7055b_0255_4a0f_a1f6_971cae2af884.slice/crio-8f62a610a1e695bf08107c170f489a2839ef6fb7bff84608c8203365c805814e WatchSource:0}: Error finding container 8f62a610a1e695bf08107c170f489a2839ef6fb7bff84608c8203365c805814e: Status 404 returned error can't find the container with id 8f62a610a1e695bf08107c170f489a2839ef6fb7bff84608c8203365c805814e
Sep 16 18:42:27.339757 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.339723    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-config\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.339878 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.339827    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpbg\" (UniqueName: \"kubernetes.io/projected/8f4f4d71-0f52-461c-be63-e0d68cdf9559-kube-api-access-jmpbg\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.339878 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.339848    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-oauth-serving-cert\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.339981 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.339876    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-trusted-ca-bundle\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.340032 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.340006    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-serving-cert\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.340120 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.340062    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-oauth-config\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.340175 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.340125    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-service-ca\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.394947 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.394906    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp"]
Sep 16 18:42:27.441357 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.441315    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpbg\" (UniqueName: \"kubernetes.io/projected/8f4f4d71-0f52-461c-be63-e0d68cdf9559-kube-api-access-jmpbg\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.441357 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.441359    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-oauth-serving-cert\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.441620 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.441382    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-trusted-ca-bundle\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.441620 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.441446    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-serving-cert\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.441620 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.441478    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-oauth-config\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.441620 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.441508    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-service-ca\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.441620 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.441541    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-config\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.442440 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.442321    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-oauth-serving-cert\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.442440 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.442425    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-config\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.442635 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.442570    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-trusted-ca-bundle\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.443166 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.443130    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-service-ca\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.444782 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.444752    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-oauth-config\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.444782 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.444776    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-serving-cert\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.460347 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.460278    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpbg\" (UniqueName: \"kubernetes.io/projected/8f4f4d71-0f52-461c-be63-e0d68cdf9559-kube-api-access-jmpbg\") pod \"console-55cff56f6d-5twn2\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") " pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.563036 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:27.562979    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:27.812610 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:42:27.812567    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67daef5_60ea_49a2_8a86_a6bf94dcebbc.slice/crio-5125ad01684534b9703036faabacaece1902b94e92fc043783922b08cda8bbee WatchSource:0}: Error finding container 5125ad01684534b9703036faabacaece1902b94e92fc043783922b08cda8bbee: Status 404 returned error can't find the container with id 5125ad01684534b9703036faabacaece1902b94e92fc043783922b08cda8bbee
Sep 16 18:42:28.101392 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:28.101244    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55cff56f6d-5twn2"]
Sep 16 18:42:28.108500 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:42:28.108454    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4f4d71_0f52_461c_be63_e0d68cdf9559.slice/crio-d2b29c52ef9e42aa308b72f2363abf898d47585643a833462657becd9e8220fd WatchSource:0}: Error finding container d2b29c52ef9e42aa308b72f2363abf898d47585643a833462657becd9e8220fd: Status 404 returned error can't find the container with id d2b29c52ef9e42aa308b72f2363abf898d47585643a833462657becd9e8220fd
Sep 16 18:42:28.277282 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:28.277234    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55cff56f6d-5twn2" event={"ID":"8f4f4d71-0f52-461c-be63-e0d68cdf9559","Type":"ContainerStarted","Data":"f70fa738756151f1413fe4a7e249ba76aff1d62b99d8b72fdbdd8a48028f0afb"}
Sep 16 18:42:28.277462 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:28.277304    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55cff56f6d-5twn2" event={"ID":"8f4f4d71-0f52-461c-be63-e0d68cdf9559","Type":"ContainerStarted","Data":"d2b29c52ef9e42aa308b72f2363abf898d47585643a833462657becd9e8220fd"}
Sep 16 18:42:28.279373 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:28.279335    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp" event={"ID":"d67daef5-60ea-49a2-8a86-a6bf94dcebbc","Type":"ContainerStarted","Data":"5125ad01684534b9703036faabacaece1902b94e92fc043783922b08cda8bbee"}
Sep 16 18:42:28.281038 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:28.280959    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr" event={"ID":"36f7055b-0255-4a0f-a1f6-971cae2af884","Type":"ContainerStarted","Data":"8f62a610a1e695bf08107c170f489a2839ef6fb7bff84608c8203365c805814e"}
Sep 16 18:42:28.283262 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:28.283226    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8" event={"ID":"bbe0642e-3461-40d3-bee5-6be6545fc3d1","Type":"ContainerStarted","Data":"77ea2df8be97df4fef4e006ecb9b7e1ecd700e95a329ff90fd9f844bb2d9ddc5"}
Sep 16 18:42:28.298340 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:28.298269    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55cff56f6d-5twn2" podStartSLOduration=1.298242388 podStartE2EDuration="1.298242388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:28.29646699 +0000 UTC m=+144.190654262" watchObservedRunningTime="2026-09-16 18:42:28.298242388 +0000 UTC m=+144.192429658"
Sep 16 18:42:28.314473 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:28.314409    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8" podStartSLOduration=1.81812866 podStartE2EDuration="3.314381692s" totalImagesPullingTime="1.496253032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:28.313805936 +0000 UTC m=+144.207993206" watchObservedRunningTime="2026-09-16 18:42:28.314381692 +0000 UTC m=+144.208568962"
Sep 16 18:42:29.498028 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:29.497977    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:30.241063 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.241025    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-886957dbf-7nb9z"]
Sep 16 18:42:30.241400 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:42:30.241373    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-886957dbf-7nb9z" podUID="490ebb65-e892-4133-8653-fe085b6f034e"
Sep 16 18:42:30.295349 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.295305    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp" event={"ID":"d67daef5-60ea-49a2-8a86-a6bf94dcebbc","Type":"ContainerStarted","Data":"144a6dc6291bea8496310bf7b30ae5d6a66acc75db7e897c4621d866bf587e84"}
Sep 16 18:42:30.295527 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.295500    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp"
Sep 16 18:42:30.297317 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.297287    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:42:30.297317 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.297304    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr" event={"ID":"36f7055b-0255-4a0f-a1f6-971cae2af884","Type":"ContainerStarted","Data":"34eba3e64df7860aa3f380734513896d916c2f193c8c70d3cd43fd3fb3cdebd0"}
Sep 16 18:42:30.301228 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.301198    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp"
Sep 16 18:42:30.305410 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.305374    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:42:30.316715 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.316658    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kbwnp" podStartSLOduration=2.651051665 podStartE2EDuration="4.31663647s" totalImagesPullingTime="1.665584805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:30.315593017 +0000 UTC m=+146.209780286" watchObservedRunningTime="2026-09-16 18:42:30.31663647 +0000 UTC m=+146.210823739"
Sep 16 18:42:30.470741 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.470692    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/490ebb65-e892-4133-8653-fe085b6f034e-trusted-ca\") pod \"490ebb65-e892-4133-8653-fe085b6f034e\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") "
Sep 16 18:42:30.470741 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.470740    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/490ebb65-e892-4133-8653-fe085b6f034e-registry-certificates\") pod \"490ebb65-e892-4133-8653-fe085b6f034e\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") "
Sep 16 18:42:30.470948 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.470779    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/490ebb65-e892-4133-8653-fe085b6f034e-installation-pull-secrets\") pod \"490ebb65-e892-4133-8653-fe085b6f034e\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") "
Sep 16 18:42:30.470948 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.470834    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/490ebb65-e892-4133-8653-fe085b6f034e-image-registry-private-configuration\") pod \"490ebb65-e892-4133-8653-fe085b6f034e\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") "
Sep 16 18:42:30.470948 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.470872    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/490ebb65-e892-4133-8653-fe085b6f034e-ca-trust-extracted\") pod \"490ebb65-e892-4133-8653-fe085b6f034e\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") "
Sep 16 18:42:30.470948 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.470898    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-bound-sa-token\") pod \"490ebb65-e892-4133-8653-fe085b6f034e\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") "
Sep 16 18:42:30.471177 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.470951    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgfrz\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-kube-api-access-cgfrz\") pod \"490ebb65-e892-4133-8653-fe085b6f034e\" (UID: \"490ebb65-e892-4133-8653-fe085b6f034e\") "
Sep 16 18:42:30.471380 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.471347    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490ebb65-e892-4133-8653-fe085b6f034e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "490ebb65-e892-4133-8653-fe085b6f034e" (UID: "490ebb65-e892-4133-8653-fe085b6f034e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:30.471947 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.471915    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/490ebb65-e892-4133-8653-fe085b6f034e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "490ebb65-e892-4133-8653-fe085b6f034e" (UID: "490ebb65-e892-4133-8653-fe085b6f034e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:30.472184 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.472162    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490ebb65-e892-4133-8653-fe085b6f034e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "490ebb65-e892-4133-8653-fe085b6f034e" (UID: "490ebb65-e892-4133-8653-fe085b6f034e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:30.473932 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.473901    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/490ebb65-e892-4133-8653-fe085b6f034e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "490ebb65-e892-4133-8653-fe085b6f034e" (UID: "490ebb65-e892-4133-8653-fe085b6f034e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:30.474028 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.473928    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-kube-api-access-cgfrz" (OuterVolumeSpecName: "kube-api-access-cgfrz") pod "490ebb65-e892-4133-8653-fe085b6f034e" (UID: "490ebb65-e892-4133-8653-fe085b6f034e"). InnerVolumeSpecName "kube-api-access-cgfrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:30.474028 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.473940    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/490ebb65-e892-4133-8653-fe085b6f034e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "490ebb65-e892-4133-8653-fe085b6f034e" (UID: "490ebb65-e892-4133-8653-fe085b6f034e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:30.474228 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.474208    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "490ebb65-e892-4133-8653-fe085b6f034e" (UID: "490ebb65-e892-4133-8653-fe085b6f034e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:30.572154 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.572124    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/490ebb65-e892-4133-8653-fe085b6f034e-trusted-ca\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:30.572154 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.572150    2552 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/490ebb65-e892-4133-8653-fe085b6f034e-registry-certificates\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:30.572154 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.572160    2552 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/490ebb65-e892-4133-8653-fe085b6f034e-installation-pull-secrets\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:30.572806 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.572171    2552 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/490ebb65-e892-4133-8653-fe085b6f034e-image-registry-private-configuration\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:30.572806 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.572180    2552 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/490ebb65-e892-4133-8653-fe085b6f034e-ca-trust-extracted\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:30.572806 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.572189    2552 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-bound-sa-token\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:30.572806 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:30.572197    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgfrz\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-kube-api-access-cgfrz\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.304216 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.304181    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-886957dbf-7nb9z"
Sep 16 18:42:31.304216 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.304184    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr" event={"ID":"36f7055b-0255-4a0f-a1f6-971cae2af884","Type":"ContainerStarted","Data":"3ed10995dad6f61d5853707f369ba136e71393af349e4ff1fad22fbeb9db0e1f"}
Sep 16 18:42:31.304446 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.304256    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr" event={"ID":"36f7055b-0255-4a0f-a1f6-971cae2af884","Type":"ContainerStarted","Data":"74d8e22373f723a05dc78f4c13d95ae41e610ff0c4843e46392ac3373e881b64"}
Sep 16 18:42:31.338836 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.338797    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-886957dbf-7nb9z"]
Sep 16 18:42:31.343218 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.343180    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-886957dbf-7nb9z"]
Sep 16 18:42:31.365948 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.365891    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-75cc585756-x9qwr" podStartSLOduration=2.317174205 podStartE2EDuration="5.365872878s" totalImagesPullingTime="3.048698673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:31.363660048 +0000 UTC m=+147.257847317" watchObservedRunningTime="2026-09-16 18:42:31.365872878 +0000 UTC m=+147.260060146"
Sep 16 18:42:31.471598 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.471559    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55cff56f6d-5twn2"]
Sep 16 18:42:31.479943 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.479900    2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/490ebb65-e892-4133-8653-fe085b6f034e-registry-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.497922 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.497883    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cd84f6f77-4qwgm"]
Sep 16 18:42:31.501360 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.501337    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.513360 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.513327    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cd84f6f77-4qwgm"]
Sep 16 18:42:31.580724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.580631    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-config\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.580724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.580669    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-oauth-serving-cert\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.580724 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.580697    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-serving-cert\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.581182 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.580784    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-oauth-config\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.581182 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.580816    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-service-ca\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.581182 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.580886    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-trusted-ca-bundle\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.581182 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.580909    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr95t\" (UniqueName: \"kubernetes.io/projected/5dbe43d3-93c8-4dad-b594-16d9d4480a27-kube-api-access-xr95t\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.681569 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.681526    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-config\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.681569 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.681573    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-oauth-serving-cert\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.681785 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.681597    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-serving-cert\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.681785 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.681745    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-oauth-config\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.681859 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.681790    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-service-ca\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.681908 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.681894    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-trusted-ca-bundle\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.681957 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.681929    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr95t\" (UniqueName: \"kubernetes.io/projected/5dbe43d3-93c8-4dad-b594-16d9d4480a27-kube-api-access-xr95t\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.682574 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.682536    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-oauth-serving-cert\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.682710 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.682554    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-config\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.682768 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.682740    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-service-ca\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.683336 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.683314    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-trusted-ca-bundle\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.685971 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.685556    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-serving-cert\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.685971 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.685684    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-oauth-config\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.691003 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.690964    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr95t\" (UniqueName: \"kubernetes.io/projected/5dbe43d3-93c8-4dad-b594-16d9d4480a27-kube-api-access-xr95t\") pod \"console-7cd84f6f77-4qwgm\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") " pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:31.810936 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:31.810889    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:32.096730 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:32.096693    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cd84f6f77-4qwgm"]
Sep 16 18:42:32.100757 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:42:32.100726    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dbe43d3_93c8_4dad_b594_16d9d4480a27.slice/crio-93db7cd0c935f9eb83744b614a990b422fcfec1483e0d9e141ae13cebee816eb WatchSource:0}: Error finding container 93db7cd0c935f9eb83744b614a990b422fcfec1483e0d9e141ae13cebee816eb: Status 404 returned error can't find the container with id 93db7cd0c935f9eb83744b614a990b422fcfec1483e0d9e141ae13cebee816eb
Sep 16 18:42:32.314400 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:32.314355    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd84f6f77-4qwgm" event={"ID":"5dbe43d3-93c8-4dad-b594-16d9d4480a27","Type":"ContainerStarted","Data":"2c602c8e591ae836752762a6782ae188b1f643c896ec9ffcf0fc3ec1d75033d4"}
Sep 16 18:42:32.314589 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:32.314459    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd84f6f77-4qwgm" event={"ID":"5dbe43d3-93c8-4dad-b594-16d9d4480a27","Type":"ContainerStarted","Data":"93db7cd0c935f9eb83744b614a990b422fcfec1483e0d9e141ae13cebee816eb"}
Sep 16 18:42:32.333109 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:32.333032    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cd84f6f77-4qwgm" podStartSLOduration=1.333013385 podStartE2EDuration="1.333013385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:32.331772458 +0000 UTC m=+148.225959740" watchObservedRunningTime="2026-09-16 18:42:32.333013385 +0000 UTC m=+148.227200654"
Sep 16 18:42:32.624558 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:32.624522    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="490ebb65-e892-4133-8653-fe085b6f034e" path="/var/lib/kubelet/pods/490ebb65-e892-4133-8653-fe085b6f034e/volumes"
Sep 16 18:42:37.563685 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:37.563630    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:41.811963 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:41.811919    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:41.812513 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:41.811983    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:41.817057 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:41.817028    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:42.359285 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:42.359255    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:42:42.412580 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:42.412537    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-95f4c55d6-jw47r"]
Sep 16 18:42:42.438871 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:42:42.438826    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-cs4tf" podUID="083ee546-65d4-43b1-9cb4-383989984db0"
Sep 16 18:42:43.357672 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:43.357638    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:42:46.094194 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:46.094159    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:46.094582 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:46.094253    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:42:47.422590 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:47.422546    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:42:47.425029 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:47.424993    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083ee546-65d4-43b1-9cb4-383989984db0-metrics-tls\") pod \"dns-default-cs4tf\" (UID: \"083ee546-65d4-43b1-9cb4-383989984db0\") " pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:42:47.561320 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:47.561277    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f92fp\""
Sep 16 18:42:47.568668 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:47.568631    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:42:47.908532 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:47.908497    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cs4tf"]
Sep 16 18:42:47.911796 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:42:47.911767    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083ee546_65d4_43b1_9cb4_383989984db0.slice/crio-3d8ecfdcf534e25435debd8df7efcff4305d2c1fa84d45a3e5250e89e326173c WatchSource:0}: Error finding container 3d8ecfdcf534e25435debd8df7efcff4305d2c1fa84d45a3e5250e89e326173c: Status 404 returned error can't find the container with id 3d8ecfdcf534e25435debd8df7efcff4305d2c1fa84d45a3e5250e89e326173c
Sep 16 18:42:48.379120 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:48.379065    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cs4tf" event={"ID":"083ee546-65d4-43b1-9cb4-383989984db0","Type":"ContainerStarted","Data":"3d8ecfdcf534e25435debd8df7efcff4305d2c1fa84d45a3e5250e89e326173c"}
Sep 16 18:42:50.389037 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:50.388990    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cs4tf" event={"ID":"083ee546-65d4-43b1-9cb4-383989984db0","Type":"ContainerStarted","Data":"a00d41a1f1748ab6063a23fad7d4f36e83ca698b29063a6644d1c71b42a3ac26"}
Sep 16 18:42:50.389473 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:50.389066    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:42:50.389473 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:50.389114    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cs4tf" event={"ID":"083ee546-65d4-43b1-9cb4-383989984db0","Type":"ContainerStarted","Data":"a965a14e6afd40f9091277bb07e037466576f8fd4e5f0d2f9fc438041593bbbb"}
Sep 16 18:42:50.405916 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:50.405862    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cs4tf" podStartSLOduration=129.760488209 podStartE2EDuration="2m11.4058424s" totalImagesPullingTime="1.645354191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:50.404871825 +0000 UTC m=+166.299059093" watchObservedRunningTime="2026-09-16 18:42:50.4058424 +0000 UTC m=+166.300029666"
Sep 16 18:42:52.233649 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.233582    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57c58dbd54-sc69h" podUID="1f92f32e-9616-474d-809d-b90ae021a853" containerName="console" containerID="cri-o://3863fc5ee530a01090dfc0ce745c128d40d5e78906a08e2c3f3230069ac086ff" gracePeriod=15
Sep 16 18:42:52.398783 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.398501    2552 generic.go:356] "Generic (PLEG): container finished" podID="1f92f32e-9616-474d-809d-b90ae021a853" containerID="3863fc5ee530a01090dfc0ce745c128d40d5e78906a08e2c3f3230069ac086ff" exitCode=0
Sep 16 18:42:52.398783 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.398583    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c58dbd54-sc69h" event={"ID":"1f92f32e-9616-474d-809d-b90ae021a853","Type":"ContainerDied","Data":"3863fc5ee530a01090dfc0ce745c128d40d5e78906a08e2c3f3230069ac086ff"}
Sep 16 18:42:52.483700 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.483621    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:52.569952 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.569913    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-console-config\") pod \"1f92f32e-9616-474d-809d-b90ae021a853\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") "
Sep 16 18:42:52.569952 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.569956    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f92f32e-9616-474d-809d-b90ae021a853-console-oauth-config\") pod \"1f92f32e-9616-474d-809d-b90ae021a853\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") "
Sep 16 18:42:52.570217 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.569999    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-trusted-ca-bundle\") pod \"1f92f32e-9616-474d-809d-b90ae021a853\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") "
Sep 16 18:42:52.570217 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.570163    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-service-ca\") pod \"1f92f32e-9616-474d-809d-b90ae021a853\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") "
Sep 16 18:42:52.570319 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.570248    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f92f32e-9616-474d-809d-b90ae021a853-console-serving-cert\") pod \"1f92f32e-9616-474d-809d-b90ae021a853\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") "
Sep 16 18:42:52.570377 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.570325    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhsb7\" (UniqueName: \"kubernetes.io/projected/1f92f32e-9616-474d-809d-b90ae021a853-kube-api-access-dhsb7\") pod \"1f92f32e-9616-474d-809d-b90ae021a853\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") "
Sep 16 18:42:52.570432 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.570375    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-oauth-serving-cert\") pod \"1f92f32e-9616-474d-809d-b90ae021a853\" (UID: \"1f92f32e-9616-474d-809d-b90ae021a853\") "
Sep 16 18:42:52.570490 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.570461    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-console-config" (OuterVolumeSpecName: "console-config") pod "1f92f32e-9616-474d-809d-b90ae021a853" (UID: "1f92f32e-9616-474d-809d-b90ae021a853"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:52.570542 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.570517    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-service-ca" (OuterVolumeSpecName: "service-ca") pod "1f92f32e-9616-474d-809d-b90ae021a853" (UID: "1f92f32e-9616-474d-809d-b90ae021a853"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:52.570542 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.570468    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1f92f32e-9616-474d-809d-b90ae021a853" (UID: "1f92f32e-9616-474d-809d-b90ae021a853"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:52.570691 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.570673    2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-console-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:52.570758 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.570700    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-trusted-ca-bundle\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:52.570758 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.570713    2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-service-ca\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:52.570855 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.570825    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1f92f32e-9616-474d-809d-b90ae021a853" (UID: "1f92f32e-9616-474d-809d-b90ae021a853"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:52.572490 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.572455    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f92f32e-9616-474d-809d-b90ae021a853-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1f92f32e-9616-474d-809d-b90ae021a853" (UID: "1f92f32e-9616-474d-809d-b90ae021a853"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:52.572490 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.572481    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f92f32e-9616-474d-809d-b90ae021a853-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1f92f32e-9616-474d-809d-b90ae021a853" (UID: "1f92f32e-9616-474d-809d-b90ae021a853"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:52.572633 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.572519    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f92f32e-9616-474d-809d-b90ae021a853-kube-api-access-dhsb7" (OuterVolumeSpecName: "kube-api-access-dhsb7") pod "1f92f32e-9616-474d-809d-b90ae021a853" (UID: "1f92f32e-9616-474d-809d-b90ae021a853"). InnerVolumeSpecName "kube-api-access-dhsb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:52.671520 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.671475    2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f92f32e-9616-474d-809d-b90ae021a853-console-oauth-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:52.671520 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.671509    2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f92f32e-9616-474d-809d-b90ae021a853-console-serving-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:52.671520 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.671521    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhsb7\" (UniqueName: \"kubernetes.io/projected/1f92f32e-9616-474d-809d-b90ae021a853-kube-api-access-dhsb7\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:52.671520 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:52.671530    2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f92f32e-9616-474d-809d-b90ae021a853-oauth-serving-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:53.405298 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:53.405263    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c58dbd54-sc69h"
Sep 16 18:42:53.405751 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:53.405263    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c58dbd54-sc69h" event={"ID":"1f92f32e-9616-474d-809d-b90ae021a853","Type":"ContainerDied","Data":"4b78e58376171f4d89ab3a8abd3cc805c04bff3fec8785a0ec4cb79c4dbe0bba"}
Sep 16 18:42:53.405751 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:53.405404    2552 scope.go:117] "RemoveContainer" containerID="3863fc5ee530a01090dfc0ce745c128d40d5e78906a08e2c3f3230069ac086ff"
Sep 16 18:42:53.422869 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:53.422831    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57c58dbd54-sc69h"]
Sep 16 18:42:53.426458 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:53.426422    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57c58dbd54-sc69h"]
Sep 16 18:42:54.624761 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:54.624723    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f92f32e-9616-474d-809d-b90ae021a853" path="/var/lib/kubelet/pods/1f92f32e-9616-474d-809d-b90ae021a853/volumes"
Sep 16 18:42:56.491202 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.491076    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55cff56f6d-5twn2" podUID="8f4f4d71-0f52-461c-be63-e0d68cdf9559" containerName="console" containerID="cri-o://f70fa738756151f1413fe4a7e249ba76aff1d62b99d8b72fdbdd8a48028f0afb" gracePeriod=15
Sep 16 18:42:56.741677 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.741599    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:56.806992 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.806950    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-service-ca\") pod \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") "
Sep 16 18:42:56.807268 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.807020    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-serving-cert\") pod \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") "
Sep 16 18:42:56.807268 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.807150    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmpbg\" (UniqueName: \"kubernetes.io/projected/8f4f4d71-0f52-461c-be63-e0d68cdf9559-kube-api-access-jmpbg\") pod \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") "
Sep 16 18:42:56.807268 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.807191    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-trusted-ca-bundle\") pod \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") "
Sep 16 18:42:56.807268 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.807214    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-oauth-serving-cert\") pod \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") "
Sep 16 18:42:56.807268 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.807253    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-config\") pod \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") "
Sep 16 18:42:56.807528 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.807280    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-oauth-config\") pod \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\" (UID: \"8f4f4d71-0f52-461c-be63-e0d68cdf9559\") "
Sep 16 18:42:56.808390 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.808332    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-service-ca" (OuterVolumeSpecName: "service-ca") pod "8f4f4d71-0f52-461c-be63-e0d68cdf9559" (UID: "8f4f4d71-0f52-461c-be63-e0d68cdf9559"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:56.808390 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.808328    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8f4f4d71-0f52-461c-be63-e0d68cdf9559" (UID: "8f4f4d71-0f52-461c-be63-e0d68cdf9559"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:56.814036 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.808666    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8f4f4d71-0f52-461c-be63-e0d68cdf9559" (UID: "8f4f4d71-0f52-461c-be63-e0d68cdf9559"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:56.814036 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.808717    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-config" (OuterVolumeSpecName: "console-config") pod "8f4f4d71-0f52-461c-be63-e0d68cdf9559" (UID: "8f4f4d71-0f52-461c-be63-e0d68cdf9559"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:56.814580 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.814183    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8f4f4d71-0f52-461c-be63-e0d68cdf9559" (UID: "8f4f4d71-0f52-461c-be63-e0d68cdf9559"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:56.814696 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.814616    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8f4f4d71-0f52-461c-be63-e0d68cdf9559" (UID: "8f4f4d71-0f52-461c-be63-e0d68cdf9559"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:56.814696 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.814628    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4f4d71-0f52-461c-be63-e0d68cdf9559-kube-api-access-jmpbg" (OuterVolumeSpecName: "kube-api-access-jmpbg") pod "8f4f4d71-0f52-461c-be63-e0d68cdf9559" (UID: "8f4f4d71-0f52-461c-be63-e0d68cdf9559"). InnerVolumeSpecName "kube-api-access-jmpbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:56.909077 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.909030    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmpbg\" (UniqueName: \"kubernetes.io/projected/8f4f4d71-0f52-461c-be63-e0d68cdf9559-kube-api-access-jmpbg\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:56.909077 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.909062    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-trusted-ca-bundle\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:56.909077 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.909071    2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-oauth-serving-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:56.909077 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.909081    2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:56.909425 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.909110    2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-oauth-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:56.909425 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.909121    2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4f4d71-0f52-461c-be63-e0d68cdf9559-service-ca\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:56.909425 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:56.909129    2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4f4d71-0f52-461c-be63-e0d68cdf9559-console-serving-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:42:57.423440 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:57.423398    2552 generic.go:356] "Generic (PLEG): container finished" podID="8f4f4d71-0f52-461c-be63-e0d68cdf9559" containerID="f70fa738756151f1413fe4a7e249ba76aff1d62b99d8b72fdbdd8a48028f0afb" exitCode=0
Sep 16 18:42:57.423664 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:57.423445    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55cff56f6d-5twn2" event={"ID":"8f4f4d71-0f52-461c-be63-e0d68cdf9559","Type":"ContainerDied","Data":"f70fa738756151f1413fe4a7e249ba76aff1d62b99d8b72fdbdd8a48028f0afb"}
Sep 16 18:42:57.423664 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:57.423461    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55cff56f6d-5twn2"
Sep 16 18:42:57.423664 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:57.423540    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55cff56f6d-5twn2" event={"ID":"8f4f4d71-0f52-461c-be63-e0d68cdf9559","Type":"ContainerDied","Data":"d2b29c52ef9e42aa308b72f2363abf898d47585643a833462657becd9e8220fd"}
Sep 16 18:42:57.423664 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:57.423564    2552 scope.go:117] "RemoveContainer" containerID="f70fa738756151f1413fe4a7e249ba76aff1d62b99d8b72fdbdd8a48028f0afb"
Sep 16 18:42:57.431697 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:57.431678    2552 scope.go:117] "RemoveContainer" containerID="f70fa738756151f1413fe4a7e249ba76aff1d62b99d8b72fdbdd8a48028f0afb"
Sep 16 18:42:57.431990 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:42:57.431969    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f70fa738756151f1413fe4a7e249ba76aff1d62b99d8b72fdbdd8a48028f0afb\": container with ID starting with f70fa738756151f1413fe4a7e249ba76aff1d62b99d8b72fdbdd8a48028f0afb not found: ID does not exist" containerID="f70fa738756151f1413fe4a7e249ba76aff1d62b99d8b72fdbdd8a48028f0afb"
Sep 16 18:42:57.432037 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:57.432000    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f70fa738756151f1413fe4a7e249ba76aff1d62b99d8b72fdbdd8a48028f0afb"} err="failed to get container status \"f70fa738756151f1413fe4a7e249ba76aff1d62b99d8b72fdbdd8a48028f0afb\": rpc error: code = NotFound desc = could not find container \"f70fa738756151f1413fe4a7e249ba76aff1d62b99d8b72fdbdd8a48028f0afb\": container with ID starting with f70fa738756151f1413fe4a7e249ba76aff1d62b99d8b72fdbdd8a48028f0afb not found: ID does not exist"
Sep 16 18:42:57.445066 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:57.445029    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55cff56f6d-5twn2"]
Sep 16 18:42:57.453755 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:57.453721    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55cff56f6d-5twn2"]
Sep 16 18:42:58.624069 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:42:58.624027    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f4f4d71-0f52-461c-be63-e0d68cdf9559" path="/var/lib/kubelet/pods/8f4f4d71-0f52-461c-be63-e0d68cdf9559/volumes"
Sep 16 18:43:00.395746 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:00.395713    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cs4tf"
Sep 16 18:43:06.100438 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:06.100400    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:43:06.104545 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:06.104510    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-54b8f9ff88-lkgg8"
Sep 16 18:43:07.433640 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.433590    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-95f4c55d6-jw47r" podUID="4d40557c-52e1-4932-af16-4c38702a2a87" containerName="console" containerID="cri-o://de6bc95a2283ab4a909eb45e7414b5a3a70978fcbf974e307f32250376a843ed" gracePeriod=15
Sep 16 18:43:07.687358 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.687273    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:43:07.809271 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.809227    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-service-ca\") pod \"4d40557c-52e1-4932-af16-4c38702a2a87\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") "
Sep 16 18:43:07.809479 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.809285    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d40557c-52e1-4932-af16-4c38702a2a87-console-serving-cert\") pod \"4d40557c-52e1-4932-af16-4c38702a2a87\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") "
Sep 16 18:43:07.809479 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.809325    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npk6k\" (UniqueName: \"kubernetes.io/projected/4d40557c-52e1-4932-af16-4c38702a2a87-kube-api-access-npk6k\") pod \"4d40557c-52e1-4932-af16-4c38702a2a87\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") "
Sep 16 18:43:07.809479 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.809423    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-console-config\") pod \"4d40557c-52e1-4932-af16-4c38702a2a87\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") "
Sep 16 18:43:07.809636 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.809493    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-oauth-serving-cert\") pod \"4d40557c-52e1-4932-af16-4c38702a2a87\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") "
Sep 16 18:43:07.809636 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.809551    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d40557c-52e1-4932-af16-4c38702a2a87-console-oauth-config\") pod \"4d40557c-52e1-4932-af16-4c38702a2a87\" (UID: \"4d40557c-52e1-4932-af16-4c38702a2a87\") "
Sep 16 18:43:07.809790 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.809762    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-service-ca" (OuterVolumeSpecName: "service-ca") pod "4d40557c-52e1-4932-af16-4c38702a2a87" (UID: "4d40557c-52e1-4932-af16-4c38702a2a87"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:07.809906 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.809795    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-console-config" (OuterVolumeSpecName: "console-config") pod "4d40557c-52e1-4932-af16-4c38702a2a87" (UID: "4d40557c-52e1-4932-af16-4c38702a2a87"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:07.809906 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.809883    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4d40557c-52e1-4932-af16-4c38702a2a87" (UID: "4d40557c-52e1-4932-af16-4c38702a2a87"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:07.811648 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.811611    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d40557c-52e1-4932-af16-4c38702a2a87-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4d40557c-52e1-4932-af16-4c38702a2a87" (UID: "4d40557c-52e1-4932-af16-4c38702a2a87"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:07.811648 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.811634    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d40557c-52e1-4932-af16-4c38702a2a87-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4d40557c-52e1-4932-af16-4c38702a2a87" (UID: "4d40557c-52e1-4932-af16-4c38702a2a87"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:07.811805 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.811619    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d40557c-52e1-4932-af16-4c38702a2a87-kube-api-access-npk6k" (OuterVolumeSpecName: "kube-api-access-npk6k") pod "4d40557c-52e1-4932-af16-4c38702a2a87" (UID: "4d40557c-52e1-4932-af16-4c38702a2a87"). InnerVolumeSpecName "kube-api-access-npk6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:07.910928 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.910881    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npk6k\" (UniqueName: \"kubernetes.io/projected/4d40557c-52e1-4932-af16-4c38702a2a87-kube-api-access-npk6k\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:43:07.910928 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.910920    2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-console-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:43:07.910928 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.910929    2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-oauth-serving-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:43:07.910928 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.910938    2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d40557c-52e1-4932-af16-4c38702a2a87-console-oauth-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:43:07.911219 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.910947    2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d40557c-52e1-4932-af16-4c38702a2a87-service-ca\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:43:07.911219 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:07.910956    2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d40557c-52e1-4932-af16-4c38702a2a87-console-serving-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:43:08.475194 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:08.475148    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d40557c-52e1-4932-af16-4c38702a2a87" containerID="de6bc95a2283ab4a909eb45e7414b5a3a70978fcbf974e307f32250376a843ed" exitCode=0
Sep 16 18:43:08.475194 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:08.475195    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95f4c55d6-jw47r"
Sep 16 18:43:08.475698 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:08.475223    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95f4c55d6-jw47r" event={"ID":"4d40557c-52e1-4932-af16-4c38702a2a87","Type":"ContainerDied","Data":"de6bc95a2283ab4a909eb45e7414b5a3a70978fcbf974e307f32250376a843ed"}
Sep 16 18:43:08.475698 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:08.475270    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95f4c55d6-jw47r" event={"ID":"4d40557c-52e1-4932-af16-4c38702a2a87","Type":"ContainerDied","Data":"c3507023d07ddc82f848a76251bd16155a290dc4462d17022cf3fa3f0f398903"}
Sep 16 18:43:08.475698 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:08.475297    2552 scope.go:117] "RemoveContainer" containerID="de6bc95a2283ab4a909eb45e7414b5a3a70978fcbf974e307f32250376a843ed"
Sep 16 18:43:08.484664 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:08.484638    2552 scope.go:117] "RemoveContainer" containerID="de6bc95a2283ab4a909eb45e7414b5a3a70978fcbf974e307f32250376a843ed"
Sep 16 18:43:08.485034 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:43:08.485014    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6bc95a2283ab4a909eb45e7414b5a3a70978fcbf974e307f32250376a843ed\": container with ID starting with de6bc95a2283ab4a909eb45e7414b5a3a70978fcbf974e307f32250376a843ed not found: ID does not exist" containerID="de6bc95a2283ab4a909eb45e7414b5a3a70978fcbf974e307f32250376a843ed"
Sep 16 18:43:08.485078 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:08.485045    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6bc95a2283ab4a909eb45e7414b5a3a70978fcbf974e307f32250376a843ed"} err="failed to get container status \"de6bc95a2283ab4a909eb45e7414b5a3a70978fcbf974e307f32250376a843ed\": rpc error: code = NotFound desc = could not find container \"de6bc95a2283ab4a909eb45e7414b5a3a70978fcbf974e307f32250376a843ed\": container with ID starting with de6bc95a2283ab4a909eb45e7414b5a3a70978fcbf974e307f32250376a843ed not found: ID does not exist"
Sep 16 18:43:08.497494 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:08.497455    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-95f4c55d6-jw47r"]
Sep 16 18:43:08.501610 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:08.501577    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-95f4c55d6-jw47r"]
Sep 16 18:43:08.624490 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:08.624454    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d40557c-52e1-4932-af16-4c38702a2a87" path="/var/lib/kubelet/pods/4d40557c-52e1-4932-af16-4c38702a2a87/volumes"
Sep 16 18:43:49.523505 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.523469    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cd4b8f6c4-srlsp"]
Sep 16 18:43:49.523991 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.523780    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d40557c-52e1-4932-af16-4c38702a2a87" containerName="console"
Sep 16 18:43:49.523991 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.523793    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d40557c-52e1-4932-af16-4c38702a2a87" containerName="console"
Sep 16 18:43:49.523991 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.523805    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f92f32e-9616-474d-809d-b90ae021a853" containerName="console"
Sep 16 18:43:49.523991 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.523811    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f92f32e-9616-474d-809d-b90ae021a853" containerName="console"
Sep 16 18:43:49.523991 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.523824    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f4f4d71-0f52-461c-be63-e0d68cdf9559" containerName="console"
Sep 16 18:43:49.523991 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.523830    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f4f4d71-0f52-461c-be63-e0d68cdf9559" containerName="console"
Sep 16 18:43:49.523991 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.523872    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d40557c-52e1-4932-af16-4c38702a2a87" containerName="console"
Sep 16 18:43:49.523991 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.523879    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f92f32e-9616-474d-809d-b90ae021a853" containerName="console"
Sep 16 18:43:49.523991 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.523885    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f4f4d71-0f52-461c-be63-e0d68cdf9559" containerName="console"
Sep 16 18:43:49.526719 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.526697    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.539749 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.539709    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd4b8f6c4-srlsp"]
Sep 16 18:43:49.544667 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.544631    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cca5920-96cf-4bfa-bc90-92eec226173c-console-oauth-config\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.544831 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.544677    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cca5920-96cf-4bfa-bc90-92eec226173c-console-serving-cert\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.544831 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.544700    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-console-config\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.544831 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.544755    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-service-ca\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.544831 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.544770    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-trusted-ca-bundle\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.544969 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.544842    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-oauth-serving-cert\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.544969 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.544888    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qg2w\" (UniqueName: \"kubernetes.io/projected/9cca5920-96cf-4bfa-bc90-92eec226173c-kube-api-access-2qg2w\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.645801 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.645751    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cca5920-96cf-4bfa-bc90-92eec226173c-console-serving-cert\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.645801 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.645798    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-console-config\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.646051 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.645919    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-service-ca\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.646051 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.645952    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-trusted-ca-bundle\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.646051 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.645995    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-oauth-serving-cert\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.646051 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.646022    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qg2w\" (UniqueName: \"kubernetes.io/projected/9cca5920-96cf-4bfa-bc90-92eec226173c-kube-api-access-2qg2w\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.646317 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.646054    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cca5920-96cf-4bfa-bc90-92eec226173c-console-oauth-config\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.646650 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.646621    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-console-config\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.646758 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.646647    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-service-ca\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.646801 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.646749    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-oauth-serving-cert\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.646901 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.646883    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-trusted-ca-bundle\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.648561 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.648529    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cca5920-96cf-4bfa-bc90-92eec226173c-console-oauth-config\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.648653 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.648616    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cca5920-96cf-4bfa-bc90-92eec226173c-console-serving-cert\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.655493 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.655461    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qg2w\" (UniqueName: \"kubernetes.io/projected/9cca5920-96cf-4bfa-bc90-92eec226173c-kube-api-access-2qg2w\") pod \"console-cd4b8f6c4-srlsp\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") " pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:49.836945 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:49.836843    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:50.113469 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:50.113424    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd4b8f6c4-srlsp"]
Sep 16 18:43:50.116148 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:43:50.116117    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cca5920_96cf_4bfa_bc90_92eec226173c.slice/crio-bcdf150796411491281ecbfc724912524916eacdc9da02539fedc69d55af049f WatchSource:0}: Error finding container bcdf150796411491281ecbfc724912524916eacdc9da02539fedc69d55af049f: Status 404 returned error can't find the container with id bcdf150796411491281ecbfc724912524916eacdc9da02539fedc69d55af049f
Sep 16 18:43:50.645107 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:50.645051    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd4b8f6c4-srlsp" event={"ID":"9cca5920-96cf-4bfa-bc90-92eec226173c","Type":"ContainerStarted","Data":"daf196edf29fa996d5958019c40c8f7471dcaeaea0826aa56ca010009b0c520e"}
Sep 16 18:43:50.645617 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:50.645132    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd4b8f6c4-srlsp" event={"ID":"9cca5920-96cf-4bfa-bc90-92eec226173c","Type":"ContainerStarted","Data":"bcdf150796411491281ecbfc724912524916eacdc9da02539fedc69d55af049f"}
Sep 16 18:43:50.665655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:50.665600    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cd4b8f6c4-srlsp" podStartSLOduration=1.6655820289999999 podStartE2EDuration="1.665582029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:50.664712533 +0000 UTC m=+226.558899813" watchObservedRunningTime="2026-09-16 18:43:50.665582029 +0000 UTC m=+226.559769299"
Sep 16 18:43:59.837298 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:59.837254    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:59.837768 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:59.837434    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:43:59.842450 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:43:59.842421    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:44:00.685984 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:00.685953    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:44:00.735414 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:00.735381    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cd84f6f77-4qwgm"]
Sep 16 18:44:25.756497 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:25.756444    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7cd84f6f77-4qwgm" podUID="5dbe43d3-93c8-4dad-b594-16d9d4480a27" containerName="console" containerID="cri-o://2c602c8e591ae836752762a6782ae188b1f643c896ec9ffcf0fc3ec1d75033d4" gracePeriod=15
Sep 16 18:44:25.995057 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:25.995021    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:44:26.138887 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.138850    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-serving-cert\") pod \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") "
Sep 16 18:44:26.139064 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.138918    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-oauth-serving-cert\") pod \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") "
Sep 16 18:44:26.139064 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.138947    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr95t\" (UniqueName: \"kubernetes.io/projected/5dbe43d3-93c8-4dad-b594-16d9d4480a27-kube-api-access-xr95t\") pod \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") "
Sep 16 18:44:26.139064 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.138967    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-trusted-ca-bundle\") pod \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") "
Sep 16 18:44:26.139064 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.138999    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-oauth-config\") pod \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") "
Sep 16 18:44:26.139064 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.139040    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-config\") pod \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") "
Sep 16 18:44:26.139354 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.139272    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-service-ca\") pod \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\" (UID: \"5dbe43d3-93c8-4dad-b594-16d9d4480a27\") "
Sep 16 18:44:26.139458 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.139431    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-config" (OuterVolumeSpecName: "console-config") pod "5dbe43d3-93c8-4dad-b594-16d9d4480a27" (UID: "5dbe43d3-93c8-4dad-b594-16d9d4480a27"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:26.139535 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.139470    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5dbe43d3-93c8-4dad-b594-16d9d4480a27" (UID: "5dbe43d3-93c8-4dad-b594-16d9d4480a27"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:26.139535 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.139431    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5dbe43d3-93c8-4dad-b594-16d9d4480a27" (UID: "5dbe43d3-93c8-4dad-b594-16d9d4480a27"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:26.139649 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.139577    2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:44:26.139649 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.139640    2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-oauth-serving-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:44:26.139749 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.139658    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-trusted-ca-bundle\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:44:26.139798 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.139779    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-service-ca" (OuterVolumeSpecName: "service-ca") pod "5dbe43d3-93c8-4dad-b594-16d9d4480a27" (UID: "5dbe43d3-93c8-4dad-b594-16d9d4480a27"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:26.141239 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.141212    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5dbe43d3-93c8-4dad-b594-16d9d4480a27" (UID: "5dbe43d3-93c8-4dad-b594-16d9d4480a27"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:26.141239 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.141221    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbe43d3-93c8-4dad-b594-16d9d4480a27-kube-api-access-xr95t" (OuterVolumeSpecName: "kube-api-access-xr95t") pod "5dbe43d3-93c8-4dad-b594-16d9d4480a27" (UID: "5dbe43d3-93c8-4dad-b594-16d9d4480a27"). InnerVolumeSpecName "kube-api-access-xr95t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:26.141405 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.141298    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5dbe43d3-93c8-4dad-b594-16d9d4480a27" (UID: "5dbe43d3-93c8-4dad-b594-16d9d4480a27"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:26.240480 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.240440    2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-serving-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:44:26.240480 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.240473    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr95t\" (UniqueName: \"kubernetes.io/projected/5dbe43d3-93c8-4dad-b594-16d9d4480a27-kube-api-access-xr95t\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:44:26.240480 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.240485    2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5dbe43d3-93c8-4dad-b594-16d9d4480a27-console-oauth-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:44:26.240718 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.240494    2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5dbe43d3-93c8-4dad-b594-16d9d4480a27-service-ca\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:44:26.783845 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.783810    2552 generic.go:356] "Generic (PLEG): container finished" podID="5dbe43d3-93c8-4dad-b594-16d9d4480a27" containerID="2c602c8e591ae836752762a6782ae188b1f643c896ec9ffcf0fc3ec1d75033d4" exitCode=0
Sep 16 18:44:26.784284 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.783856    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd84f6f77-4qwgm"
Sep 16 18:44:26.784284 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.783892    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd84f6f77-4qwgm" event={"ID":"5dbe43d3-93c8-4dad-b594-16d9d4480a27","Type":"ContainerDied","Data":"2c602c8e591ae836752762a6782ae188b1f643c896ec9ffcf0fc3ec1d75033d4"}
Sep 16 18:44:26.784284 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.783945    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd84f6f77-4qwgm" event={"ID":"5dbe43d3-93c8-4dad-b594-16d9d4480a27","Type":"ContainerDied","Data":"93db7cd0c935f9eb83744b614a990b422fcfec1483e0d9e141ae13cebee816eb"}
Sep 16 18:44:26.784284 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.783966    2552 scope.go:117] "RemoveContainer" containerID="2c602c8e591ae836752762a6782ae188b1f643c896ec9ffcf0fc3ec1d75033d4"
Sep 16 18:44:26.792280 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.792255    2552 scope.go:117] "RemoveContainer" containerID="2c602c8e591ae836752762a6782ae188b1f643c896ec9ffcf0fc3ec1d75033d4"
Sep 16 18:44:26.792638 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:44:26.792613    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c602c8e591ae836752762a6782ae188b1f643c896ec9ffcf0fc3ec1d75033d4\": container with ID starting with 2c602c8e591ae836752762a6782ae188b1f643c896ec9ffcf0fc3ec1d75033d4 not found: ID does not exist" containerID="2c602c8e591ae836752762a6782ae188b1f643c896ec9ffcf0fc3ec1d75033d4"
Sep 16 18:44:26.792682 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.792648    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c602c8e591ae836752762a6782ae188b1f643c896ec9ffcf0fc3ec1d75033d4"} err="failed to get container status \"2c602c8e591ae836752762a6782ae188b1f643c896ec9ffcf0fc3ec1d75033d4\": rpc error: code = NotFound desc = could not find container \"2c602c8e591ae836752762a6782ae188b1f643c896ec9ffcf0fc3ec1d75033d4\": container with ID starting with 2c602c8e591ae836752762a6782ae188b1f643c896ec9ffcf0fc3ec1d75033d4 not found: ID does not exist"
Sep 16 18:44:26.803279 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.803241    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cd84f6f77-4qwgm"]
Sep 16 18:44:26.807505 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:26.807475    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cd84f6f77-4qwgm"]
Sep 16 18:44:28.624600 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:44:28.624564    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dbe43d3-93c8-4dad-b594-16d9d4480a27" path="/var/lib/kubelet/pods/5dbe43d3-93c8-4dad-b594-16d9d4480a27/volumes"
Sep 16 18:45:04.499297 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:04.499266    2552 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:45:12.069622 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.069584    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"]
Sep 16 18:45:12.071696 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.069880    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dbe43d3-93c8-4dad-b594-16d9d4480a27" containerName="console"
Sep 16 18:45:12.071696 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.069893    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dbe43d3-93c8-4dad-b594-16d9d4480a27" containerName="console"
Sep 16 18:45:12.071696 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.069955    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dbe43d3-93c8-4dad-b594-16d9d4480a27" containerName="console"
Sep 16 18:45:12.072623 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.072601    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"
Sep 16 18:45:12.075356 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.075325    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gnw4f\""
Sep 16 18:45:12.075472 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.075326    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:45:12.076253 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.076237    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:45:12.084584 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.084540    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"]
Sep 16 18:45:12.198275 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.198233    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f77423f5-0eb9-4f64-a662-fa4a897d6181-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw\" (UID: \"f77423f5-0eb9-4f64-a662-fa4a897d6181\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"
Sep 16 18:45:12.198466 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.198295    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f77423f5-0eb9-4f64-a662-fa4a897d6181-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw\" (UID: \"f77423f5-0eb9-4f64-a662-fa4a897d6181\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"
Sep 16 18:45:12.198466 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.198325    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zfvx\" (UniqueName: \"kubernetes.io/projected/f77423f5-0eb9-4f64-a662-fa4a897d6181-kube-api-access-5zfvx\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw\" (UID: \"f77423f5-0eb9-4f64-a662-fa4a897d6181\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"
Sep 16 18:45:12.298838 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.298795    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f77423f5-0eb9-4f64-a662-fa4a897d6181-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw\" (UID: \"f77423f5-0eb9-4f64-a662-fa4a897d6181\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"
Sep 16 18:45:12.299022 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.298846    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zfvx\" (UniqueName: \"kubernetes.io/projected/f77423f5-0eb9-4f64-a662-fa4a897d6181-kube-api-access-5zfvx\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw\" (UID: \"f77423f5-0eb9-4f64-a662-fa4a897d6181\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"
Sep 16 18:45:12.299022 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.298898    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f77423f5-0eb9-4f64-a662-fa4a897d6181-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw\" (UID: \"f77423f5-0eb9-4f64-a662-fa4a897d6181\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"
Sep 16 18:45:12.299239 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.299210    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f77423f5-0eb9-4f64-a662-fa4a897d6181-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw\" (UID: \"f77423f5-0eb9-4f64-a662-fa4a897d6181\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"
Sep 16 18:45:12.299317 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.299276    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f77423f5-0eb9-4f64-a662-fa4a897d6181-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw\" (UID: \"f77423f5-0eb9-4f64-a662-fa4a897d6181\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"
Sep 16 18:45:12.309149 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.309115    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zfvx\" (UniqueName: \"kubernetes.io/projected/f77423f5-0eb9-4f64-a662-fa4a897d6181-kube-api-access-5zfvx\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw\" (UID: \"f77423f5-0eb9-4f64-a662-fa4a897d6181\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"
Sep 16 18:45:12.381783 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.381743    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"
Sep 16 18:45:12.659922 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.659819    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"]
Sep 16 18:45:12.662780 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:45:12.662751    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf77423f5_0eb9_4f64_a662_fa4a897d6181.slice/crio-837f8f815b228f452f7448c99d61f84fcf0c5a810b82dccacb2e52deee55dbda WatchSource:0}: Error finding container 837f8f815b228f452f7448c99d61f84fcf0c5a810b82dccacb2e52deee55dbda: Status 404 returned error can't find the container with id 837f8f815b228f452f7448c99d61f84fcf0c5a810b82dccacb2e52deee55dbda
Sep 16 18:45:12.664781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.664761    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:45:12.961908 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:12.961816    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw" event={"ID":"f77423f5-0eb9-4f64-a662-fa4a897d6181","Type":"ContainerStarted","Data":"837f8f815b228f452f7448c99d61f84fcf0c5a810b82dccacb2e52deee55dbda"}
Sep 16 18:45:18.997705 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:18.997621    2552 generic.go:356] "Generic (PLEG): container finished" podID="f77423f5-0eb9-4f64-a662-fa4a897d6181" containerID="9a6d1e376a3fa8a90aff1250bca6a1967aac91fdd2daf1232d7c4655ee5b1c26" exitCode=0
Sep 16 18:45:18.997705 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:18.997688    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw" event={"ID":"f77423f5-0eb9-4f64-a662-fa4a897d6181","Type":"ContainerDied","Data":"9a6d1e376a3fa8a90aff1250bca6a1967aac91fdd2daf1232d7c4655ee5b1c26"}
Sep 16 18:45:22.011662 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:22.011628    2552 generic.go:356] "Generic (PLEG): container finished" podID="f77423f5-0eb9-4f64-a662-fa4a897d6181" containerID="92b046836af8bcfdf7aedee761c5b4b98dcdbfa5bc5ebc3138182841adbb7e32" exitCode=0
Sep 16 18:45:22.012143 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:22.011677    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw" event={"ID":"f77423f5-0eb9-4f64-a662-fa4a897d6181","Type":"ContainerDied","Data":"92b046836af8bcfdf7aedee761c5b4b98dcdbfa5bc5ebc3138182841adbb7e32"}
Sep 16 18:45:29.043932 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:29.043889    2552 generic.go:356] "Generic (PLEG): container finished" podID="f77423f5-0eb9-4f64-a662-fa4a897d6181" containerID="e0d5bbd3f4981228bd24f64db5a36a2ade4713d67938e84a648fda28bdd34d4e" exitCode=0
Sep 16 18:45:29.044336 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:29.043955    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw" event={"ID":"f77423f5-0eb9-4f64-a662-fa4a897d6181","Type":"ContainerDied","Data":"e0d5bbd3f4981228bd24f64db5a36a2ade4713d67938e84a648fda28bdd34d4e"}
Sep 16 18:45:30.174841 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:30.174807    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"
Sep 16 18:45:30.259787 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:30.259744    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zfvx\" (UniqueName: \"kubernetes.io/projected/f77423f5-0eb9-4f64-a662-fa4a897d6181-kube-api-access-5zfvx\") pod \"f77423f5-0eb9-4f64-a662-fa4a897d6181\" (UID: \"f77423f5-0eb9-4f64-a662-fa4a897d6181\") "
Sep 16 18:45:30.259963 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:30.259800    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f77423f5-0eb9-4f64-a662-fa4a897d6181-bundle\") pod \"f77423f5-0eb9-4f64-a662-fa4a897d6181\" (UID: \"f77423f5-0eb9-4f64-a662-fa4a897d6181\") "
Sep 16 18:45:30.259963 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:30.259824    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f77423f5-0eb9-4f64-a662-fa4a897d6181-util\") pod \"f77423f5-0eb9-4f64-a662-fa4a897d6181\" (UID: \"f77423f5-0eb9-4f64-a662-fa4a897d6181\") "
Sep 16 18:45:30.260494 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:30.260463    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f77423f5-0eb9-4f64-a662-fa4a897d6181-bundle" (OuterVolumeSpecName: "bundle") pod "f77423f5-0eb9-4f64-a662-fa4a897d6181" (UID: "f77423f5-0eb9-4f64-a662-fa4a897d6181"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:30.262084 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:30.262051    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f77423f5-0eb9-4f64-a662-fa4a897d6181-kube-api-access-5zfvx" (OuterVolumeSpecName: "kube-api-access-5zfvx") pod "f77423f5-0eb9-4f64-a662-fa4a897d6181" (UID: "f77423f5-0eb9-4f64-a662-fa4a897d6181"). InnerVolumeSpecName "kube-api-access-5zfvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:30.264229 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:30.264192    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f77423f5-0eb9-4f64-a662-fa4a897d6181-util" (OuterVolumeSpecName: "util") pod "f77423f5-0eb9-4f64-a662-fa4a897d6181" (UID: "f77423f5-0eb9-4f64-a662-fa4a897d6181"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:30.361393 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:30.361292    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zfvx\" (UniqueName: \"kubernetes.io/projected/f77423f5-0eb9-4f64-a662-fa4a897d6181-kube-api-access-5zfvx\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:45:30.361393 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:30.361327    2552 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f77423f5-0eb9-4f64-a662-fa4a897d6181-bundle\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:45:30.361393 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:30.361337    2552 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f77423f5-0eb9-4f64-a662-fa4a897d6181-util\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:45:31.053605 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:31.053515    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw"
Sep 16 18:45:31.053764 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:31.053516    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvtxw" event={"ID":"f77423f5-0eb9-4f64-a662-fa4a897d6181","Type":"ContainerDied","Data":"837f8f815b228f452f7448c99d61f84fcf0c5a810b82dccacb2e52deee55dbda"}
Sep 16 18:45:31.053764 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:31.053647    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="837f8f815b228f452f7448c99d61f84fcf0c5a810b82dccacb2e52deee55dbda"
Sep 16 18:45:34.412648 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.412610    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n"]
Sep 16 18:45:34.413037 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.412882    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f77423f5-0eb9-4f64-a662-fa4a897d6181" containerName="util"
Sep 16 18:45:34.413037 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.412893    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f77423f5-0eb9-4f64-a662-fa4a897d6181" containerName="util"
Sep 16 18:45:34.413037 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.412906    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f77423f5-0eb9-4f64-a662-fa4a897d6181" containerName="pull"
Sep 16 18:45:34.413037 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.412912    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f77423f5-0eb9-4f64-a662-fa4a897d6181" containerName="pull"
Sep 16 18:45:34.413037 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.412924    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f77423f5-0eb9-4f64-a662-fa4a897d6181" containerName="extract"
Sep 16 18:45:34.413037 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.412930    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f77423f5-0eb9-4f64-a662-fa4a897d6181" containerName="extract"
Sep 16 18:45:34.413037 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.412979    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f77423f5-0eb9-4f64-a662-fa4a897d6181" containerName="extract"
Sep 16 18:45:34.416270 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.416248    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n"
Sep 16 18:45:34.418704 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.418675    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:45:34.418833 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.418800    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:45:34.418833 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.418800    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:45:34.418965 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.418943    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-pf9wr\""
Sep 16 18:45:34.431594 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.431559    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n"]
Sep 16 18:45:34.497626 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.497588    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/a10a0a70-4070-4329-9e5f-1f8b35302394-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-5m69n\" (UID: \"a10a0a70-4070-4329-9e5f-1f8b35302394\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n"
Sep 16 18:45:34.497814 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.497651    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hg65\" (UniqueName: \"kubernetes.io/projected/a10a0a70-4070-4329-9e5f-1f8b35302394-kube-api-access-8hg65\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-5m69n\" (UID: \"a10a0a70-4070-4329-9e5f-1f8b35302394\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n"
Sep 16 18:45:34.599049 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.599004    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/a10a0a70-4070-4329-9e5f-1f8b35302394-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-5m69n\" (UID: \"a10a0a70-4070-4329-9e5f-1f8b35302394\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n"
Sep 16 18:45:34.599243 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.599058    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hg65\" (UniqueName: \"kubernetes.io/projected/a10a0a70-4070-4329-9e5f-1f8b35302394-kube-api-access-8hg65\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-5m69n\" (UID: \"a10a0a70-4070-4329-9e5f-1f8b35302394\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n"
Sep 16 18:45:34.601506 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.601475    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/a10a0a70-4070-4329-9e5f-1f8b35302394-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-5m69n\" (UID: \"a10a0a70-4070-4329-9e5f-1f8b35302394\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n"
Sep 16 18:45:34.609127 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.609080    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hg65\" (UniqueName: \"kubernetes.io/projected/a10a0a70-4070-4329-9e5f-1f8b35302394-kube-api-access-8hg65\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-5m69n\" (UID: \"a10a0a70-4070-4329-9e5f-1f8b35302394\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n"
Sep 16 18:45:34.726987 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:34.726893    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n"
Sep 16 18:45:35.004463 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:35.004376    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n"]
Sep 16 18:45:35.008526 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:45:35.008484    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10a0a70_4070_4329_9e5f_1f8b35302394.slice/crio-73ddeb4b53612c890d5f59c7b59baa5001c950a9414debd73a00b305f1a53397 WatchSource:0}: Error finding container 73ddeb4b53612c890d5f59c7b59baa5001c950a9414debd73a00b305f1a53397: Status 404 returned error can't find the container with id 73ddeb4b53612c890d5f59c7b59baa5001c950a9414debd73a00b305f1a53397
Sep 16 18:45:35.070802 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:35.070766    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n" event={"ID":"a10a0a70-4070-4329-9e5f-1f8b35302394","Type":"ContainerStarted","Data":"73ddeb4b53612c890d5f59c7b59baa5001c950a9414debd73a00b305f1a53397"}
Sep 16 18:45:39.091305 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.091264    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n" event={"ID":"a10a0a70-4070-4329-9e5f-1f8b35302394","Type":"ContainerStarted","Data":"8584268d9e08039159ebf39c0acb1bde7e54a2e000070e37db82ba3879f4cde7"}
Sep 16 18:45:39.091888 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.091376    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n"
Sep 16 18:45:39.119068 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.119009    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n" podStartSLOduration=1.817386387 podStartE2EDuration="5.118988964s" totalImagesPullingTime="3.301602577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:39.117826316 +0000 UTC m=+335.012013587" watchObservedRunningTime="2026-09-16 18:45:39.118988964 +0000 UTC m=+335.013176233"
Sep 16 18:45:39.482295 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.482251    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"]
Sep 16 18:45:39.484614 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.484594    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"
Sep 16 18:45:39.486832 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.486803    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 16 18:45:39.486832 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.486807    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-d2kf4\""
Sep 16 18:45:39.487077 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.486971    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:45:39.495461 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.495427    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"]
Sep 16 18:45:39.635556 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.635510    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-dtpkh"]
Sep 16 18:45:39.637357 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.637323    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t9p9\" (UniqueName: \"kubernetes.io/projected/c3340fbb-e447-47dd-9e6a-6d3f8b2c3308-kube-api-access-9t9p9\") pod \"keda-metrics-apiserver-77857b7b98-jxpwb\" (UID: \"c3340fbb-e447-47dd-9e6a-6d3f8b2c3308\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"
Sep 16 18:45:39.637503 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.637433    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c3340fbb-e447-47dd-9e6a-6d3f8b2c3308-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-jxpwb\" (UID: \"c3340fbb-e447-47dd-9e6a-6d3f8b2c3308\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"
Sep 16 18:45:39.637503 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.637473    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c3340fbb-e447-47dd-9e6a-6d3f8b2c3308-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jxpwb\" (UID: \"c3340fbb-e447-47dd-9e6a-6d3f8b2c3308\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"
Sep 16 18:45:39.637753 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.637734    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-dtpkh"
Sep 16 18:45:39.639905 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.639873    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 18:45:39.647198 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.647168    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-dtpkh"]
Sep 16 18:45:39.738688 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.738590    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9b224df0-c5ce-4f8b-a7d7-3700a04b1042-certificates\") pod \"keda-admission-6c4d864b86-dtpkh\" (UID: \"9b224df0-c5ce-4f8b-a7d7-3700a04b1042\") " pod="openshift-keda/keda-admission-6c4d864b86-dtpkh"
Sep 16 18:45:39.738688 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.738632    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58xn\" (UniqueName: \"kubernetes.io/projected/9b224df0-c5ce-4f8b-a7d7-3700a04b1042-kube-api-access-d58xn\") pod \"keda-admission-6c4d864b86-dtpkh\" (UID: \"9b224df0-c5ce-4f8b-a7d7-3700a04b1042\") " pod="openshift-keda/keda-admission-6c4d864b86-dtpkh"
Sep 16 18:45:39.738688 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.738663    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c3340fbb-e447-47dd-9e6a-6d3f8b2c3308-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-jxpwb\" (UID: \"c3340fbb-e447-47dd-9e6a-6d3f8b2c3308\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"
Sep 16 18:45:39.738913 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.738744    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c3340fbb-e447-47dd-9e6a-6d3f8b2c3308-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jxpwb\" (UID: \"c3340fbb-e447-47dd-9e6a-6d3f8b2c3308\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"
Sep 16 18:45:39.738913 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.738817    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t9p9\" (UniqueName: \"kubernetes.io/projected/c3340fbb-e447-47dd-9e6a-6d3f8b2c3308-kube-api-access-9t9p9\") pod \"keda-metrics-apiserver-77857b7b98-jxpwb\" (UID: \"c3340fbb-e447-47dd-9e6a-6d3f8b2c3308\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"
Sep 16 18:45:39.738913 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:45:39.738850    2552 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 16 18:45:39.738913 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:45:39.738877    2552 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb: secret "keda-metrics-apiserver-certs" not found
Sep 16 18:45:39.739063 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:45:39.738948    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3340fbb-e447-47dd-9e6a-6d3f8b2c3308-certificates podName:c3340fbb-e447-47dd-9e6a-6d3f8b2c3308 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:40.238925634 +0000 UTC m=+336.133112884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c3340fbb-e447-47dd-9e6a-6d3f8b2c3308-certificates") pod "keda-metrics-apiserver-77857b7b98-jxpwb" (UID: "c3340fbb-e447-47dd-9e6a-6d3f8b2c3308") : secret "keda-metrics-apiserver-certs" not found
Sep 16 18:45:39.739162 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.739140    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c3340fbb-e447-47dd-9e6a-6d3f8b2c3308-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-jxpwb\" (UID: \"c3340fbb-e447-47dd-9e6a-6d3f8b2c3308\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"
Sep 16 18:45:39.748311 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.748274    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t9p9\" (UniqueName: \"kubernetes.io/projected/c3340fbb-e447-47dd-9e6a-6d3f8b2c3308-kube-api-access-9t9p9\") pod \"keda-metrics-apiserver-77857b7b98-jxpwb\" (UID: \"c3340fbb-e447-47dd-9e6a-6d3f8b2c3308\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"
Sep 16 18:45:39.840082 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.840036    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9b224df0-c5ce-4f8b-a7d7-3700a04b1042-certificates\") pod \"keda-admission-6c4d864b86-dtpkh\" (UID: \"9b224df0-c5ce-4f8b-a7d7-3700a04b1042\") " pod="openshift-keda/keda-admission-6c4d864b86-dtpkh"
Sep 16 18:45:39.840082 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.840083    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58xn\" (UniqueName: \"kubernetes.io/projected/9b224df0-c5ce-4f8b-a7d7-3700a04b1042-kube-api-access-d58xn\") pod \"keda-admission-6c4d864b86-dtpkh\" (UID: \"9b224df0-c5ce-4f8b-a7d7-3700a04b1042\") " pod="openshift-keda/keda-admission-6c4d864b86-dtpkh"
Sep 16 18:45:39.842977 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.842943    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9b224df0-c5ce-4f8b-a7d7-3700a04b1042-certificates\") pod \"keda-admission-6c4d864b86-dtpkh\" (UID: \"9b224df0-c5ce-4f8b-a7d7-3700a04b1042\") " pod="openshift-keda/keda-admission-6c4d864b86-dtpkh"
Sep 16 18:45:39.849401 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.849376    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58xn\" (UniqueName: \"kubernetes.io/projected/9b224df0-c5ce-4f8b-a7d7-3700a04b1042-kube-api-access-d58xn\") pod \"keda-admission-6c4d864b86-dtpkh\" (UID: \"9b224df0-c5ce-4f8b-a7d7-3700a04b1042\") " pod="openshift-keda/keda-admission-6c4d864b86-dtpkh"
Sep 16 18:45:39.949206 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:39.949163    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-dtpkh"
Sep 16 18:45:40.238514 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:40.238427    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-dtpkh"]
Sep 16 18:45:40.242276 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:45:40.242246    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b224df0_c5ce_4f8b_a7d7_3700a04b1042.slice/crio-615e693f2dcf9bf1463dc80232bc9e628c14dbc22bde8e8783becc643d8a6e63 WatchSource:0}: Error finding container 615e693f2dcf9bf1463dc80232bc9e628c14dbc22bde8e8783becc643d8a6e63: Status 404 returned error can't find the container with id 615e693f2dcf9bf1463dc80232bc9e628c14dbc22bde8e8783becc643d8a6e63
Sep 16 18:45:40.245229 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:40.245202    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c3340fbb-e447-47dd-9e6a-6d3f8b2c3308-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jxpwb\" (UID: \"c3340fbb-e447-47dd-9e6a-6d3f8b2c3308\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"
Sep 16 18:45:40.248536 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:40.248377    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c3340fbb-e447-47dd-9e6a-6d3f8b2c3308-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jxpwb\" (UID: \"c3340fbb-e447-47dd-9e6a-6d3f8b2c3308\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"
Sep 16 18:45:40.396302 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:40.396266    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"
Sep 16 18:45:40.671435 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:40.671350    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"]
Sep 16 18:45:40.674293 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:45:40.674264    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3340fbb_e447_47dd_9e6a_6d3f8b2c3308.slice/crio-9b84f9c48a62bf1ee41808b619ccd55c1b614eef7cf525d6d8324745e75e258d WatchSource:0}: Error finding container 9b84f9c48a62bf1ee41808b619ccd55c1b614eef7cf525d6d8324745e75e258d: Status 404 returned error can't find the container with id 9b84f9c48a62bf1ee41808b619ccd55c1b614eef7cf525d6d8324745e75e258d
Sep 16 18:45:41.101583 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:41.101486    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb" event={"ID":"c3340fbb-e447-47dd-9e6a-6d3f8b2c3308","Type":"ContainerStarted","Data":"9b84f9c48a62bf1ee41808b619ccd55c1b614eef7cf525d6d8324745e75e258d"}
Sep 16 18:45:41.103011 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:41.102978    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-dtpkh" event={"ID":"9b224df0-c5ce-4f8b-a7d7-3700a04b1042","Type":"ContainerStarted","Data":"615e693f2dcf9bf1463dc80232bc9e628c14dbc22bde8e8783becc643d8a6e63"}
Sep 16 18:45:42.108916 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:42.108835    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-dtpkh" event={"ID":"9b224df0-c5ce-4f8b-a7d7-3700a04b1042","Type":"ContainerStarted","Data":"b67128d8e68491fd27f768e1fc8bf666c9e4f925fa62275078db24db5c22a1f7"}
Sep 16 18:45:42.108916 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:42.108904    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-dtpkh"
Sep 16 18:45:42.125791 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:42.125738    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-dtpkh" podStartSLOduration=1.6242593360000002 podStartE2EDuration="3.125721759s" totalImagesPullingTime="1.501462423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:42.124822001 +0000 UTC m=+338.019009270" watchObservedRunningTime="2026-09-16 18:45:42.125721759 +0000 UTC m=+338.019909029"
Sep 16 18:45:44.118991 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:44.118949    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb" event={"ID":"c3340fbb-e447-47dd-9e6a-6d3f8b2c3308","Type":"ContainerStarted","Data":"ec70e215078bdbf25a67e3ed9a7547af69ea9608f737e3b468446a4dd1f1e7f9"}
Sep 16 18:45:44.119385 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:44.119063    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"
Sep 16 18:45:44.142858 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:44.142546    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb" podStartSLOduration=2.529319467 podStartE2EDuration="5.142524585s" totalImagesPullingTime="2.613205118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:44.142156983 +0000 UTC m=+340.036344251" watchObservedRunningTime="2026-09-16 18:45:44.142524585 +0000 UTC m=+340.036711856"
Sep 16 18:45:55.127994 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:45:55.127962    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jxpwb"
Sep 16 18:46:00.098882 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:00.098853    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5m69n"
Sep 16 18:46:03.116166 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:03.116136    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-dtpkh"
Sep 16 18:46:56.474298 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.474262    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"]
Sep 16 18:46:56.476421 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.476401    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:46:56.479057 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.479030    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 16 18:46:56.479206 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.479033    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:46:56.479873 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.479855    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:46:56.479960 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.479855    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-wk76n\""
Sep 16 18:46:56.487237 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.487209    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"]
Sep 16 18:46:56.501264 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.501231    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-d8qkl"]
Sep 16 18:46:56.503488 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.503468    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-d8qkl"
Sep 16 18:46:56.505730 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.505705    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:46:56.505921 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.505730    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-g47rj\""
Sep 16 18:46:56.517110 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.515321    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-d8qkl"]
Sep 16 18:46:56.571536 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.571492    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7297ae2a-6069-4a8a-aadb-9ebaf57ddc90-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-mvmcq\" (UID: \"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:46:56.571716 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.571547    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dd615c57-c80e-4893-906d-355e4aa634cf-data\") pod \"seaweedfs-64568bcd49-d8qkl\" (UID: \"dd615c57-c80e-4893-906d-355e4aa634cf\") " pod="kserve/seaweedfs-64568bcd49-d8qkl"
Sep 16 18:46:56.571716 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.571631    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfdh4\" (UniqueName: \"kubernetes.io/projected/dd615c57-c80e-4893-906d-355e4aa634cf-kube-api-access-gfdh4\") pod \"seaweedfs-64568bcd49-d8qkl\" (UID: \"dd615c57-c80e-4893-906d-355e4aa634cf\") " pod="kserve/seaweedfs-64568bcd49-d8qkl"
Sep 16 18:46:56.571716 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.571671    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fdgg\" (UniqueName: \"kubernetes.io/projected/7297ae2a-6069-4a8a-aadb-9ebaf57ddc90-kube-api-access-9fdgg\") pod \"llmisvc-controller-manager-645f9c8fd6-mvmcq\" (UID: \"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:46:56.672779 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.672730    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dd615c57-c80e-4893-906d-355e4aa634cf-data\") pod \"seaweedfs-64568bcd49-d8qkl\" (UID: \"dd615c57-c80e-4893-906d-355e4aa634cf\") " pod="kserve/seaweedfs-64568bcd49-d8qkl"
Sep 16 18:46:56.672997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.672819    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfdh4\" (UniqueName: \"kubernetes.io/projected/dd615c57-c80e-4893-906d-355e4aa634cf-kube-api-access-gfdh4\") pod \"seaweedfs-64568bcd49-d8qkl\" (UID: \"dd615c57-c80e-4893-906d-355e4aa634cf\") " pod="kserve/seaweedfs-64568bcd49-d8qkl"
Sep 16 18:46:56.672997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.672847    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fdgg\" (UniqueName: \"kubernetes.io/projected/7297ae2a-6069-4a8a-aadb-9ebaf57ddc90-kube-api-access-9fdgg\") pod \"llmisvc-controller-manager-645f9c8fd6-mvmcq\" (UID: \"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:46:56.672997 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.672897    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7297ae2a-6069-4a8a-aadb-9ebaf57ddc90-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-mvmcq\" (UID: \"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:46:56.673276 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.673169    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dd615c57-c80e-4893-906d-355e4aa634cf-data\") pod \"seaweedfs-64568bcd49-d8qkl\" (UID: \"dd615c57-c80e-4893-906d-355e4aa634cf\") " pod="kserve/seaweedfs-64568bcd49-d8qkl"
Sep 16 18:46:56.675459 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.675431    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7297ae2a-6069-4a8a-aadb-9ebaf57ddc90-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-mvmcq\" (UID: \"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:46:56.681806 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.681767    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfdh4\" (UniqueName: \"kubernetes.io/projected/dd615c57-c80e-4893-906d-355e4aa634cf-kube-api-access-gfdh4\") pod \"seaweedfs-64568bcd49-d8qkl\" (UID: \"dd615c57-c80e-4893-906d-355e4aa634cf\") " pod="kserve/seaweedfs-64568bcd49-d8qkl"
Sep 16 18:46:56.681933 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.681898    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fdgg\" (UniqueName: \"kubernetes.io/projected/7297ae2a-6069-4a8a-aadb-9ebaf57ddc90-kube-api-access-9fdgg\") pod \"llmisvc-controller-manager-645f9c8fd6-mvmcq\" (UID: \"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:46:56.787752 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.787646    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:46:56.815781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:56.815746    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-d8qkl"
Sep 16 18:46:57.067350 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:57.067257    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"]
Sep 16 18:46:57.070235 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:46:57.070208    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7297ae2a_6069_4a8a_aadb_9ebaf57ddc90.slice/crio-53181c84a11df9d7c4c47308af7a15a04fe6f5343885240580aa19490bbfe663 WatchSource:0}: Error finding container 53181c84a11df9d7c4c47308af7a15a04fe6f5343885240580aa19490bbfe663: Status 404 returned error can't find the container with id 53181c84a11df9d7c4c47308af7a15a04fe6f5343885240580aa19490bbfe663
Sep 16 18:46:57.101137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:57.101077    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-d8qkl"]
Sep 16 18:46:57.104160 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:46:57.104132    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd615c57_c80e_4893_906d_355e4aa634cf.slice/crio-5f3fdf530ab11f98832a048eafaca6388797fa3a5b764f83e97ddf2ca19ba749 WatchSource:0}: Error finding container 5f3fdf530ab11f98832a048eafaca6388797fa3a5b764f83e97ddf2ca19ba749: Status 404 returned error can't find the container with id 5f3fdf530ab11f98832a048eafaca6388797fa3a5b764f83e97ddf2ca19ba749
Sep 16 18:46:57.436267 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:57.436216    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerStarted","Data":"53181c84a11df9d7c4c47308af7a15a04fe6f5343885240580aa19490bbfe663"}
Sep 16 18:46:57.437762 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:46:57.437729    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-d8qkl" event={"ID":"dd615c57-c80e-4893-906d-355e4aa634cf","Type":"ContainerStarted","Data":"5f3fdf530ab11f98832a048eafaca6388797fa3a5b764f83e97ddf2ca19ba749"}
Sep 16 18:47:01.459433 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:01.459393    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerStarted","Data":"bda2004f11373864e55c88d711297c357e1d880b408368ad6e8873a5c20f60a2"}
Sep 16 18:47:01.459859 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:01.459465    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:47:01.460994 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:01.460967    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-d8qkl" event={"ID":"dd615c57-c80e-4893-906d-355e4aa634cf","Type":"ContainerStarted","Data":"a41ffc735fd019bc2744e706e0ffe0a144aeed2f8c52fd213c33f74ab0d425ed"}
Sep 16 18:47:01.461132 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:01.461012    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-d8qkl"
Sep 16 18:47:01.476578 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:01.476518    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podStartSLOduration=2.0347257 podStartE2EDuration="5.476499086s" totalImagesPullingTime="3.441773386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:01.474821731 +0000 UTC m=+417.369009002" watchObservedRunningTime="2026-09-16 18:47:01.476499086 +0000 UTC m=+417.370686355"
Sep 16 18:47:01.490670 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:01.490598    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-d8qkl" podStartSLOduration=1.988737699 podStartE2EDuration="5.490582355s" totalImagesPullingTime="3.501844656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:01.489869868 +0000 UTC m=+417.384057138" watchObservedRunningTime="2026-09-16 18:47:01.490582355 +0000 UTC m=+417.384769624"
Sep 16 18:47:07.467285 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:07.467244    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-d8qkl"
Sep 16 18:47:14.124121 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.124065    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-648d6c6bd5-tgvzq"]
Sep 16 18:47:14.128893 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.128858    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.141102 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.141053    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-648d6c6bd5-tgvzq"]
Sep 16 18:47:14.223419 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.223357    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2eff5a46-e5d8-4e51-956d-09843653654e-oauth-serving-cert\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.223419 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.223424    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2eff5a46-e5d8-4e51-956d-09843653654e-console-config\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.223651 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.223489    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eff5a46-e5d8-4e51-956d-09843653654e-trusted-ca-bundle\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.223651 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.223527    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2eff5a46-e5d8-4e51-956d-09843653654e-console-oauth-config\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.223651 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.223554    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2eff5a46-e5d8-4e51-956d-09843653654e-console-serving-cert\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.223651 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.223637    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2eff5a46-e5d8-4e51-956d-09843653654e-service-ca\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.223778 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.223672    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wldrr\" (UniqueName: \"kubernetes.io/projected/2eff5a46-e5d8-4e51-956d-09843653654e-kube-api-access-wldrr\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.324305 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.324244    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2eff5a46-e5d8-4e51-956d-09843653654e-service-ca\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.324305 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.324306    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wldrr\" (UniqueName: \"kubernetes.io/projected/2eff5a46-e5d8-4e51-956d-09843653654e-kube-api-access-wldrr\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.324576 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.324363    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2eff5a46-e5d8-4e51-956d-09843653654e-oauth-serving-cert\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.324576 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.324395    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2eff5a46-e5d8-4e51-956d-09843653654e-console-config\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.324576 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.324526    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eff5a46-e5d8-4e51-956d-09843653654e-trusted-ca-bundle\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.324731 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.324589    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2eff5a46-e5d8-4e51-956d-09843653654e-console-oauth-config\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.324731 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.324629    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2eff5a46-e5d8-4e51-956d-09843653654e-console-serving-cert\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.325183 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.325154    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2eff5a46-e5d8-4e51-956d-09843653654e-console-config\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.325183 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.325173    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2eff5a46-e5d8-4e51-956d-09843653654e-oauth-serving-cert\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.325374 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.325154    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2eff5a46-e5d8-4e51-956d-09843653654e-service-ca\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.325430 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.325393    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eff5a46-e5d8-4e51-956d-09843653654e-trusted-ca-bundle\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.327171 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.327144    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2eff5a46-e5d8-4e51-956d-09843653654e-console-oauth-config\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.327262 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.327235    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2eff5a46-e5d8-4e51-956d-09843653654e-console-serving-cert\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.339923 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.339889    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wldrr\" (UniqueName: \"kubernetes.io/projected/2eff5a46-e5d8-4e51-956d-09843653654e-kube-api-access-wldrr\") pod \"console-648d6c6bd5-tgvzq\" (UID: \"2eff5a46-e5d8-4e51-956d-09843653654e\") " pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.438577 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.438482    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:14.713772 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:14.713691    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-648d6c6bd5-tgvzq"]
Sep 16 18:47:14.717365 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:47:14.717337    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eff5a46_e5d8_4e51_956d_09843653654e.slice/crio-6dfe11eac47e5e151467b8ed5c20c73a7b5588cd21708cc608a957df42a8146b WatchSource:0}: Error finding container 6dfe11eac47e5e151467b8ed5c20c73a7b5588cd21708cc608a957df42a8146b: Status 404 returned error can't find the container with id 6dfe11eac47e5e151467b8ed5c20c73a7b5588cd21708cc608a957df42a8146b
Sep 16 18:47:15.524660 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:15.524626    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648d6c6bd5-tgvzq" event={"ID":"2eff5a46-e5d8-4e51-956d-09843653654e","Type":"ContainerStarted","Data":"c48f56dff3fd19d896cefc21f8e706ddf7317185b8049b4993734043859b5793"}
Sep 16 18:47:15.525051 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:15.524677    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648d6c6bd5-tgvzq" event={"ID":"2eff5a46-e5d8-4e51-956d-09843653654e","Type":"ContainerStarted","Data":"6dfe11eac47e5e151467b8ed5c20c73a7b5588cd21708cc608a957df42a8146b"}
Sep 16 18:47:15.549364 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:15.549308    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-648d6c6bd5-tgvzq" podStartSLOduration=1.549291542 podStartE2EDuration="1.549291542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:15.549073664 +0000 UTC m=+431.443260935" watchObservedRunningTime="2026-09-16 18:47:15.549291542 +0000 UTC m=+431.443478822"
Sep 16 18:47:24.438868 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:24.438803    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:24.439340 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:24.438901    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:24.444081 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:24.444052    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:24.567881 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:24.567847    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-648d6c6bd5-tgvzq"
Sep 16 18:47:24.614554 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:24.614518    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd4b8f6c4-srlsp"]
Sep 16 18:47:32.467544 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:32.467449    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:47:49.637558 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:49.637495    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cd4b8f6c4-srlsp" podUID="9cca5920-96cf-4bfa-bc90-92eec226173c" containerName="console" containerID="cri-o://daf196edf29fa996d5958019c40c8f7471dcaeaea0826aa56ca010009b0c520e" gracePeriod=15
Sep 16 18:47:49.889748 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:49.889676    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:47:50.042042 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.041999    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-service-ca\") pod \"9cca5920-96cf-4bfa-bc90-92eec226173c\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") "
Sep 16 18:47:50.042245 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.042075    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cca5920-96cf-4bfa-bc90-92eec226173c-console-oauth-config\") pod \"9cca5920-96cf-4bfa-bc90-92eec226173c\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") "
Sep 16 18:47:50.042245 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.042168    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-console-config\") pod \"9cca5920-96cf-4bfa-bc90-92eec226173c\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") "
Sep 16 18:47:50.042245 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.042220    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-trusted-ca-bundle\") pod \"9cca5920-96cf-4bfa-bc90-92eec226173c\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") "
Sep 16 18:47:50.042423 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.042249    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cca5920-96cf-4bfa-bc90-92eec226173c-console-serving-cert\") pod \"9cca5920-96cf-4bfa-bc90-92eec226173c\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") "
Sep 16 18:47:50.042423 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.042279    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-oauth-serving-cert\") pod \"9cca5920-96cf-4bfa-bc90-92eec226173c\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") "
Sep 16 18:47:50.042423 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.042326    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qg2w\" (UniqueName: \"kubernetes.io/projected/9cca5920-96cf-4bfa-bc90-92eec226173c-kube-api-access-2qg2w\") pod \"9cca5920-96cf-4bfa-bc90-92eec226173c\" (UID: \"9cca5920-96cf-4bfa-bc90-92eec226173c\") "
Sep 16 18:47:50.043008 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.042890    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-service-ca" (OuterVolumeSpecName: "service-ca") pod "9cca5920-96cf-4bfa-bc90-92eec226173c" (UID: "9cca5920-96cf-4bfa-bc90-92eec226173c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:50.043008 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.042944    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-console-config" (OuterVolumeSpecName: "console-config") pod "9cca5920-96cf-4bfa-bc90-92eec226173c" (UID: "9cca5920-96cf-4bfa-bc90-92eec226173c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:50.043008 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.042956    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9cca5920-96cf-4bfa-bc90-92eec226173c" (UID: "9cca5920-96cf-4bfa-bc90-92eec226173c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:50.043426 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.043390    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9cca5920-96cf-4bfa-bc90-92eec226173c" (UID: "9cca5920-96cf-4bfa-bc90-92eec226173c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:50.050804 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.050764    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cca5920-96cf-4bfa-bc90-92eec226173c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9cca5920-96cf-4bfa-bc90-92eec226173c" (UID: "9cca5920-96cf-4bfa-bc90-92eec226173c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:50.050804 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.050779    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cca5920-96cf-4bfa-bc90-92eec226173c-kube-api-access-2qg2w" (OuterVolumeSpecName: "kube-api-access-2qg2w") pod "9cca5920-96cf-4bfa-bc90-92eec226173c" (UID: "9cca5920-96cf-4bfa-bc90-92eec226173c"). InnerVolumeSpecName "kube-api-access-2qg2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:50.051047 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.050780    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cca5920-96cf-4bfa-bc90-92eec226173c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9cca5920-96cf-4bfa-bc90-92eec226173c" (UID: "9cca5920-96cf-4bfa-bc90-92eec226173c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:50.143715 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.143614    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qg2w\" (UniqueName: \"kubernetes.io/projected/9cca5920-96cf-4bfa-bc90-92eec226173c-kube-api-access-2qg2w\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:47:50.143715 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.143652    2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-service-ca\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:47:50.143715 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.143663    2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cca5920-96cf-4bfa-bc90-92eec226173c-console-oauth-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:47:50.143715 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.143672    2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-console-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:47:50.143715 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.143681    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-trusted-ca-bundle\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:47:50.143715 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.143690    2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cca5920-96cf-4bfa-bc90-92eec226173c-console-serving-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:47:50.143715 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.143698    2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cca5920-96cf-4bfa-bc90-92eec226173c-oauth-serving-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:47:50.675100 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.675052    2552 generic.go:356] "Generic (PLEG): container finished" podID="9cca5920-96cf-4bfa-bc90-92eec226173c" containerID="daf196edf29fa996d5958019c40c8f7471dcaeaea0826aa56ca010009b0c520e" exitCode=0
Sep 16 18:47:50.675566 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.675123    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd4b8f6c4-srlsp"
Sep 16 18:47:50.675566 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.675135    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd4b8f6c4-srlsp" event={"ID":"9cca5920-96cf-4bfa-bc90-92eec226173c","Type":"ContainerDied","Data":"daf196edf29fa996d5958019c40c8f7471dcaeaea0826aa56ca010009b0c520e"}
Sep 16 18:47:50.675566 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.675194    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd4b8f6c4-srlsp" event={"ID":"9cca5920-96cf-4bfa-bc90-92eec226173c","Type":"ContainerDied","Data":"bcdf150796411491281ecbfc724912524916eacdc9da02539fedc69d55af049f"}
Sep 16 18:47:50.675566 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.675226    2552 scope.go:117] "RemoveContainer" containerID="daf196edf29fa996d5958019c40c8f7471dcaeaea0826aa56ca010009b0c520e"
Sep 16 18:47:50.684040 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.684017    2552 scope.go:117] "RemoveContainer" containerID="daf196edf29fa996d5958019c40c8f7471dcaeaea0826aa56ca010009b0c520e"
Sep 16 18:47:50.684433 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:47:50.684409    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf196edf29fa996d5958019c40c8f7471dcaeaea0826aa56ca010009b0c520e\": container with ID starting with daf196edf29fa996d5958019c40c8f7471dcaeaea0826aa56ca010009b0c520e not found: ID does not exist" containerID="daf196edf29fa996d5958019c40c8f7471dcaeaea0826aa56ca010009b0c520e"
Sep 16 18:47:50.684548 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.684440    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf196edf29fa996d5958019c40c8f7471dcaeaea0826aa56ca010009b0c520e"} err="failed to get container status \"daf196edf29fa996d5958019c40c8f7471dcaeaea0826aa56ca010009b0c520e\": rpc error: code = NotFound desc = could not find container \"daf196edf29fa996d5958019c40c8f7471dcaeaea0826aa56ca010009b0c520e\": container with ID starting with daf196edf29fa996d5958019c40c8f7471dcaeaea0826aa56ca010009b0c520e not found: ID does not exist"
Sep 16 18:47:50.693825 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.693792    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd4b8f6c4-srlsp"]
Sep 16 18:47:50.698166 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:50.698113    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cd4b8f6c4-srlsp"]
Sep 16 18:47:52.625843 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:47:52.625805    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cca5920-96cf-4bfa-bc90-92eec226173c" path="/var/lib/kubelet/pods/9cca5920-96cf-4bfa-bc90-92eec226173c/volumes"
Sep 16 18:48:13.937814 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:13.937775    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-f9dg5"]
Sep 16 18:48:13.938241 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:13.938126    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cca5920-96cf-4bfa-bc90-92eec226173c" containerName="console"
Sep 16 18:48:13.938241 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:13.938138    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cca5920-96cf-4bfa-bc90-92eec226173c" containerName="console"
Sep 16 18:48:13.938241 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:13.938207    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cca5920-96cf-4bfa-bc90-92eec226173c" containerName="console"
Sep 16 18:48:13.941028 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:13.941004    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-f9dg5"
Sep 16 18:48:13.944883 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:13.944848    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 18:48:13.945057 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:13.944848    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 18:48:13.945057 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:13.944932    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-b4n5h\""
Sep 16 18:48:13.957894 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:13.957859    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-f9dg5"]
Sep 16 18:48:14.028570 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:14.028526    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64085754-3c26-43b2-b8cf-60ac4140c852-metrics-certs\") pod \"odh-model-controller-7b966d7984-f9dg5\" (UID: \"64085754-3c26-43b2-b8cf-60ac4140c852\") " pod="kserve/odh-model-controller-7b966d7984-f9dg5"
Sep 16 18:48:14.028786 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:14.028610    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djbnj\" (UniqueName: \"kubernetes.io/projected/64085754-3c26-43b2-b8cf-60ac4140c852-kube-api-access-djbnj\") pod \"odh-model-controller-7b966d7984-f9dg5\" (UID: \"64085754-3c26-43b2-b8cf-60ac4140c852\") " pod="kserve/odh-model-controller-7b966d7984-f9dg5"
Sep 16 18:48:14.028786 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:14.028708    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64085754-3c26-43b2-b8cf-60ac4140c852-cert\") pod \"odh-model-controller-7b966d7984-f9dg5\" (UID: \"64085754-3c26-43b2-b8cf-60ac4140c852\") " pod="kserve/odh-model-controller-7b966d7984-f9dg5"
Sep 16 18:48:14.129861 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:14.129819    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djbnj\" (UniqueName: \"kubernetes.io/projected/64085754-3c26-43b2-b8cf-60ac4140c852-kube-api-access-djbnj\") pod \"odh-model-controller-7b966d7984-f9dg5\" (UID: \"64085754-3c26-43b2-b8cf-60ac4140c852\") " pod="kserve/odh-model-controller-7b966d7984-f9dg5"
Sep 16 18:48:14.130061 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:14.129886    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64085754-3c26-43b2-b8cf-60ac4140c852-cert\") pod \"odh-model-controller-7b966d7984-f9dg5\" (UID: \"64085754-3c26-43b2-b8cf-60ac4140c852\") " pod="kserve/odh-model-controller-7b966d7984-f9dg5"
Sep 16 18:48:14.130061 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:14.129908    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64085754-3c26-43b2-b8cf-60ac4140c852-metrics-certs\") pod \"odh-model-controller-7b966d7984-f9dg5\" (UID: \"64085754-3c26-43b2-b8cf-60ac4140c852\") " pod="kserve/odh-model-controller-7b966d7984-f9dg5"
Sep 16 18:48:14.130061 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:48:14.130048    2552 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 18:48:14.130229 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:48:14.130167    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64085754-3c26-43b2-b8cf-60ac4140c852-cert podName:64085754-3c26-43b2-b8cf-60ac4140c852 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:14.630143409 +0000 UTC m=+490.524330660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/64085754-3c26-43b2-b8cf-60ac4140c852-cert") pod "odh-model-controller-7b966d7984-f9dg5" (UID: "64085754-3c26-43b2-b8cf-60ac4140c852") : secret "odh-model-controller-webhook-cert" not found
Sep 16 18:48:14.132636 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:14.132601    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64085754-3c26-43b2-b8cf-60ac4140c852-metrics-certs\") pod \"odh-model-controller-7b966d7984-f9dg5\" (UID: \"64085754-3c26-43b2-b8cf-60ac4140c852\") " pod="kserve/odh-model-controller-7b966d7984-f9dg5"
Sep 16 18:48:14.140584 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:14.140548    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djbnj\" (UniqueName: \"kubernetes.io/projected/64085754-3c26-43b2-b8cf-60ac4140c852-kube-api-access-djbnj\") pod \"odh-model-controller-7b966d7984-f9dg5\" (UID: \"64085754-3c26-43b2-b8cf-60ac4140c852\") " pod="kserve/odh-model-controller-7b966d7984-f9dg5"
Sep 16 18:48:14.634177 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:14.634139    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64085754-3c26-43b2-b8cf-60ac4140c852-cert\") pod \"odh-model-controller-7b966d7984-f9dg5\" (UID: \"64085754-3c26-43b2-b8cf-60ac4140c852\") " pod="kserve/odh-model-controller-7b966d7984-f9dg5"
Sep 16 18:48:14.636574 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:14.636539    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64085754-3c26-43b2-b8cf-60ac4140c852-cert\") pod \"odh-model-controller-7b966d7984-f9dg5\" (UID: \"64085754-3c26-43b2-b8cf-60ac4140c852\") " pod="kserve/odh-model-controller-7b966d7984-f9dg5"
Sep 16 18:48:14.851912 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:14.851868    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-f9dg5"
Sep 16 18:48:15.123935 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:15.123898    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-f9dg5"]
Sep 16 18:48:15.127309 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:48:15.127271    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64085754_3c26_43b2_b8cf_60ac4140c852.slice/crio-00eafc2e171c8eb61387d4a552a1dc558795883900b5f9ba37171a725a7e469d WatchSource:0}: Error finding container 00eafc2e171c8eb61387d4a552a1dc558795883900b5f9ba37171a725a7e469d: Status 404 returned error can't find the container with id 00eafc2e171c8eb61387d4a552a1dc558795883900b5f9ba37171a725a7e469d
Sep 16 18:48:15.791632 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:15.791589    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-f9dg5" event={"ID":"64085754-3c26-43b2-b8cf-60ac4140c852","Type":"ContainerStarted","Data":"00eafc2e171c8eb61387d4a552a1dc558795883900b5f9ba37171a725a7e469d"}
Sep 16 18:48:18.807593 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:18.807553    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-f9dg5" event={"ID":"64085754-3c26-43b2-b8cf-60ac4140c852","Type":"ContainerStarted","Data":"1badae355a6c078b5a0a43c51a5a20668dca160226b33639bcde85ba3f2580f5"}
Sep 16 18:48:18.808078 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:18.807663    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-f9dg5"
Sep 16 18:48:18.827172 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:18.827116    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-f9dg5" podStartSLOduration=3.102874698 podStartE2EDuration="5.827079912s" totalImagesPullingTime="2.724205214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:18.824675777 +0000 UTC m=+494.718863037" watchObservedRunningTime="2026-09-16 18:48:18.827079912 +0000 UTC m=+494.721267182"
Sep 16 18:48:29.814932 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:48:29.814898    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-f9dg5"
Sep 16 18:49:02.005435 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:02.005399    2552 generic.go:356] "Generic (PLEG): container finished" podID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90" containerID="bda2004f11373864e55c88d711297c357e1d880b408368ad6e8873a5c20f60a2" exitCode=1
Sep 16 18:49:02.005849 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:02.005472    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerDied","Data":"bda2004f11373864e55c88d711297c357e1d880b408368ad6e8873a5c20f60a2"}
Sep 16 18:49:02.005849 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:02.005810    2552 scope.go:117] "RemoveContainer" containerID="bda2004f11373864e55c88d711297c357e1d880b408368ad6e8873a5c20f60a2"
Sep 16 18:49:02.465788 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:02.465744    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:49:03.012147 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:03.012081    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerStarted","Data":"d68b3274b8fdf645dec2837b9a238739792e8cb06aee5f85ed72cb783bfbe247"}
Sep 16 18:49:03.012533 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:03.012245    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:49:34.018613 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:34.018579    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:49:50.806641 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:50.806601    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"]
Sep 16 18:49:50.809208 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:50.809187    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"
Sep 16 18:49:50.811231 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:50.811207    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 16 18:49:50.820451 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:50.820421    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"]
Sep 16 18:49:50.944938 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:50.944897    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07e4074d-d9af-4dc6-a871-852a2aca418f-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-ltfzg\" (UID: \"07e4074d-d9af-4dc6-a871-852a2aca418f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"
Sep 16 18:49:50.945169 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:50.945018    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d79v4\" (UniqueName: \"kubernetes.io/projected/07e4074d-d9af-4dc6-a871-852a2aca418f-kube-api-access-d79v4\") pod \"seaweedfs-tls-custom-6644bf5d6c-ltfzg\" (UID: \"07e4074d-d9af-4dc6-a871-852a2aca418f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"
Sep 16 18:49:51.046474 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:51.046428    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07e4074d-d9af-4dc6-a871-852a2aca418f-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-ltfzg\" (UID: \"07e4074d-d9af-4dc6-a871-852a2aca418f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"
Sep 16 18:49:51.046655 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:51.046528    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d79v4\" (UniqueName: \"kubernetes.io/projected/07e4074d-d9af-4dc6-a871-852a2aca418f-kube-api-access-d79v4\") pod \"seaweedfs-tls-custom-6644bf5d6c-ltfzg\" (UID: \"07e4074d-d9af-4dc6-a871-852a2aca418f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"
Sep 16 18:49:51.046844 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:51.046823    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07e4074d-d9af-4dc6-a871-852a2aca418f-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-ltfzg\" (UID: \"07e4074d-d9af-4dc6-a871-852a2aca418f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"
Sep 16 18:49:51.066672 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:51.066590    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d79v4\" (UniqueName: \"kubernetes.io/projected/07e4074d-d9af-4dc6-a871-852a2aca418f-kube-api-access-d79v4\") pod \"seaweedfs-tls-custom-6644bf5d6c-ltfzg\" (UID: \"07e4074d-d9af-4dc6-a871-852a2aca418f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"
Sep 16 18:49:51.118425 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:51.118390    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"
Sep 16 18:49:51.388962 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:51.388894    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"]
Sep 16 18:49:51.391993 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:49:51.391964    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e4074d_d9af_4dc6_a871_852a2aca418f.slice/crio-f3dae668204a467c92c93214d2343c70b3f84d254d24f6b1b169d238344643b6 WatchSource:0}: Error finding container f3dae668204a467c92c93214d2343c70b3f84d254d24f6b1b169d238344643b6: Status 404 returned error can't find the container with id f3dae668204a467c92c93214d2343c70b3f84d254d24f6b1b169d238344643b6
Sep 16 18:49:52.225157 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:52.225112    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg" event={"ID":"07e4074d-d9af-4dc6-a871-852a2aca418f","Type":"ContainerStarted","Data":"b7088faa15aa3c6ee6b18e09180dd6e75480a88885959109dcac8f00db8037f0"}
Sep 16 18:49:52.225615 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:52.225183    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg" event={"ID":"07e4074d-d9af-4dc6-a871-852a2aca418f","Type":"ContainerStarted","Data":"f3dae668204a467c92c93214d2343c70b3f84d254d24f6b1b169d238344643b6"}
Sep 16 18:49:52.241796 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:52.241732    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg" podStartSLOduration=1.938734559 podStartE2EDuration="2.241712766s" totalImagesPullingTime="302.978207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:52.240081863 +0000 UTC m=+588.134269152" watchObservedRunningTime="2026-09-16 18:49:52.241712766 +0000 UTC m=+588.135900034"
Sep 16 18:49:53.073569 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:53.073530    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"]
Sep 16 18:49:54.234596 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:54.234545    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg" podUID="07e4074d-d9af-4dc6-a871-852a2aca418f" containerName="seaweedfs-tls-custom" containerID="cri-o://b7088faa15aa3c6ee6b18e09180dd6e75480a88885959109dcac8f00db8037f0" gracePeriod=30
Sep 16 18:49:55.478365 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:55.478332    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"
Sep 16 18:49:55.481722 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:55.481693    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07e4074d-d9af-4dc6-a871-852a2aca418f-data\") pod \"07e4074d-d9af-4dc6-a871-852a2aca418f\" (UID: \"07e4074d-d9af-4dc6-a871-852a2aca418f\") "
Sep 16 18:49:55.481801 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:55.481752    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d79v4\" (UniqueName: \"kubernetes.io/projected/07e4074d-d9af-4dc6-a871-852a2aca418f-kube-api-access-d79v4\") pod \"07e4074d-d9af-4dc6-a871-852a2aca418f\" (UID: \"07e4074d-d9af-4dc6-a871-852a2aca418f\") "
Sep 16 18:49:55.482775 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:55.482726    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e4074d-d9af-4dc6-a871-852a2aca418f-data" (OuterVolumeSpecName: "data") pod "07e4074d-d9af-4dc6-a871-852a2aca418f" (UID: "07e4074d-d9af-4dc6-a871-852a2aca418f"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:55.484010 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:55.483983    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e4074d-d9af-4dc6-a871-852a2aca418f-kube-api-access-d79v4" (OuterVolumeSpecName: "kube-api-access-d79v4") pod "07e4074d-d9af-4dc6-a871-852a2aca418f" (UID: "07e4074d-d9af-4dc6-a871-852a2aca418f"). InnerVolumeSpecName "kube-api-access-d79v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:55.582909 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:55.582810    2552 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/07e4074d-d9af-4dc6-a871-852a2aca418f-data\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:49:55.582909 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:55.582843    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d79v4\" (UniqueName: \"kubernetes.io/projected/07e4074d-d9af-4dc6-a871-852a2aca418f-kube-api-access-d79v4\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:49:56.249909 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:56.249032    2552 generic.go:356] "Generic (PLEG): container finished" podID="07e4074d-d9af-4dc6-a871-852a2aca418f" containerID="b7088faa15aa3c6ee6b18e09180dd6e75480a88885959109dcac8f00db8037f0" exitCode=0
Sep 16 18:49:56.249909 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:56.249118    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg" event={"ID":"07e4074d-d9af-4dc6-a871-852a2aca418f","Type":"ContainerDied","Data":"b7088faa15aa3c6ee6b18e09180dd6e75480a88885959109dcac8f00db8037f0"}
Sep 16 18:49:56.249909 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:56.249166    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg" event={"ID":"07e4074d-d9af-4dc6-a871-852a2aca418f","Type":"ContainerDied","Data":"f3dae668204a467c92c93214d2343c70b3f84d254d24f6b1b169d238344643b6"}
Sep 16 18:49:56.249909 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:56.249192    2552 scope.go:117] "RemoveContainer" containerID="b7088faa15aa3c6ee6b18e09180dd6e75480a88885959109dcac8f00db8037f0"
Sep 16 18:49:56.249909 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:56.249347    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"
Sep 16 18:49:56.260481 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:56.260457    2552 scope.go:117] "RemoveContainer" containerID="b7088faa15aa3c6ee6b18e09180dd6e75480a88885959109dcac8f00db8037f0"
Sep 16 18:49:56.260898 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:49:56.260877    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7088faa15aa3c6ee6b18e09180dd6e75480a88885959109dcac8f00db8037f0\": container with ID starting with b7088faa15aa3c6ee6b18e09180dd6e75480a88885959109dcac8f00db8037f0 not found: ID does not exist" containerID="b7088faa15aa3c6ee6b18e09180dd6e75480a88885959109dcac8f00db8037f0"
Sep 16 18:49:56.260948 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:56.260910    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7088faa15aa3c6ee6b18e09180dd6e75480a88885959109dcac8f00db8037f0"} err="failed to get container status \"b7088faa15aa3c6ee6b18e09180dd6e75480a88885959109dcac8f00db8037f0\": rpc error: code = NotFound desc = could not find container \"b7088faa15aa3c6ee6b18e09180dd6e75480a88885959109dcac8f00db8037f0\": container with ID starting with b7088faa15aa3c6ee6b18e09180dd6e75480a88885959109dcac8f00db8037f0 not found: ID does not exist"
Sep 16 18:49:56.271693 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:56.271657    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"]
Sep 16 18:49:56.277824 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:56.277791    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ltfzg"]
Sep 16 18:49:56.625730 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:49:56.625697    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07e4074d-d9af-4dc6-a871-852a2aca418f" path="/var/lib/kubelet/pods/07e4074d-d9af-4dc6-a871-852a2aca418f/volumes"
Sep 16 18:50:12.780843 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.780807    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9"]
Sep 16 18:50:12.783068 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.781165    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e4074d-d9af-4dc6-a871-852a2aca418f" containerName="seaweedfs-tls-custom"
Sep 16 18:50:12.783068 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.781178    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e4074d-d9af-4dc6-a871-852a2aca418f" containerName="seaweedfs-tls-custom"
Sep 16 18:50:12.783068 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.781241    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e4074d-d9af-4dc6-a871-852a2aca418f" containerName="seaweedfs-tls-custom"
Sep 16 18:50:12.783961 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.783943    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9"
Sep 16 18:50:12.786495 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.786468    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 16 18:50:12.786634 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.786468    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 16 18:50:12.791033 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.791005    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9"]
Sep 16 18:50:12.821755 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.821702    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/1a810dd1-218b-4452-bf08-0ddb51fca795-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-vcrv9\" (UID: \"1a810dd1-218b-4452-bf08-0ddb51fca795\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9"
Sep 16 18:50:12.821755 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.821750    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7qwp\" (UniqueName: \"kubernetes.io/projected/1a810dd1-218b-4452-bf08-0ddb51fca795-kube-api-access-p7qwp\") pod \"seaweedfs-tls-serving-67b7477c6d-vcrv9\" (UID: \"1a810dd1-218b-4452-bf08-0ddb51fca795\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9"
Sep 16 18:50:12.821968 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.821893    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a810dd1-218b-4452-bf08-0ddb51fca795-data\") pod \"seaweedfs-tls-serving-67b7477c6d-vcrv9\" (UID: \"1a810dd1-218b-4452-bf08-0ddb51fca795\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9"
Sep 16 18:50:12.922621 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.922560    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a810dd1-218b-4452-bf08-0ddb51fca795-data\") pod \"seaweedfs-tls-serving-67b7477c6d-vcrv9\" (UID: \"1a810dd1-218b-4452-bf08-0ddb51fca795\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9"
Sep 16 18:50:12.922621 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.922621    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/1a810dd1-218b-4452-bf08-0ddb51fca795-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-vcrv9\" (UID: \"1a810dd1-218b-4452-bf08-0ddb51fca795\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9"
Sep 16 18:50:12.922876 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.922736    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7qwp\" (UniqueName: \"kubernetes.io/projected/1a810dd1-218b-4452-bf08-0ddb51fca795-kube-api-access-p7qwp\") pod \"seaweedfs-tls-serving-67b7477c6d-vcrv9\" (UID: \"1a810dd1-218b-4452-bf08-0ddb51fca795\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9"
Sep 16 18:50:12.922976 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.922954    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a810dd1-218b-4452-bf08-0ddb51fca795-data\") pod \"seaweedfs-tls-serving-67b7477c6d-vcrv9\" (UID: \"1a810dd1-218b-4452-bf08-0ddb51fca795\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9"
Sep 16 18:50:12.925068 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.925044    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/1a810dd1-218b-4452-bf08-0ddb51fca795-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-vcrv9\" (UID: \"1a810dd1-218b-4452-bf08-0ddb51fca795\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9"
Sep 16 18:50:12.932343 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:12.932318    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7qwp\" (UniqueName: \"kubernetes.io/projected/1a810dd1-218b-4452-bf08-0ddb51fca795-kube-api-access-p7qwp\") pod \"seaweedfs-tls-serving-67b7477c6d-vcrv9\" (UID: \"1a810dd1-218b-4452-bf08-0ddb51fca795\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9"
Sep 16 18:50:13.094139 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:13.094015    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9"
Sep 16 18:50:13.368557 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:13.368477    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9"]
Sep 16 18:50:13.371359 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:50:13.371325    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a810dd1_218b_4452_bf08_0ddb51fca795.slice/crio-01a08d4510c366cb80b074a1a293bdd18fa933de6000b31cf434b433bef3cfa1 WatchSource:0}: Error finding container 01a08d4510c366cb80b074a1a293bdd18fa933de6000b31cf434b433bef3cfa1: Status 404 returned error can't find the container with id 01a08d4510c366cb80b074a1a293bdd18fa933de6000b31cf434b433bef3cfa1
Sep 16 18:50:13.372741 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:13.372716    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:50:14.332790 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:14.332741    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9" event={"ID":"1a810dd1-218b-4452-bf08-0ddb51fca795","Type":"ContainerStarted","Data":"9fef1fd75c5eb880adc9ddb68bf46e0084bed9d1dbe505d4f08c825ff3cd1c62"}
Sep 16 18:50:14.333291 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:14.332833    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9" event={"ID":"1a810dd1-218b-4452-bf08-0ddb51fca795","Type":"ContainerStarted","Data":"01a08d4510c366cb80b074a1a293bdd18fa933de6000b31cf434b433bef3cfa1"}
Sep 16 18:50:14.350720 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:14.350660    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-vcrv9" podStartSLOduration=2.098906695 podStartE2EDuration="2.350642017s" totalImagesPullingTime="251.735322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:14.348968041 +0000 UTC m=+610.243155310" watchObservedRunningTime="2026-09-16 18:50:14.350642017 +0000 UTC m=+610.244829285"
Sep 16 18:50:54.401752 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.401711    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"]
Sep 16 18:50:54.406177 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.406147    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.408625 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.408585    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:50:54.408803 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.408780    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:50:54.408900 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.408814    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 18:50:54.408900 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.408835    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 16 18:50:54.408900 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.408868    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 16 18:50:54.408900 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.408822    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:50:54.408900 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.408872    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:50:54.417265 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.417236    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"]
Sep 16 18:50:54.489836 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.489789    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/289fb560-9995-4533-af6a-4347ccc0da5b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zk96k\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.490009 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.489850    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/289fb560-9995-4533-af6a-4347ccc0da5b-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zk96k\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.490009 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.489963    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/289fb560-9995-4533-af6a-4347ccc0da5b-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zk96k\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.490121 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.490076    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzc84\" (UniqueName: \"kubernetes.io/projected/289fb560-9995-4533-af6a-4347ccc0da5b-kube-api-access-nzc84\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zk96k\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.490162 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.490136    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/289fb560-9995-4533-af6a-4347ccc0da5b-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zk96k\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.591351 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.591304    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/289fb560-9995-4533-af6a-4347ccc0da5b-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zk96k\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.591547 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.591368    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzc84\" (UniqueName: \"kubernetes.io/projected/289fb560-9995-4533-af6a-4347ccc0da5b-kube-api-access-nzc84\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zk96k\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.591547 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.591386    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/289fb560-9995-4533-af6a-4347ccc0da5b-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zk96k\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.591547 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.591413    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/289fb560-9995-4533-af6a-4347ccc0da5b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zk96k\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.591547 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.591445    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/289fb560-9995-4533-af6a-4347ccc0da5b-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zk96k\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.591880 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.591852    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/289fb560-9995-4533-af6a-4347ccc0da5b-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zk96k\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.592154 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.592132    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/289fb560-9995-4533-af6a-4347ccc0da5b-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zk96k\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.592238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.592191    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/289fb560-9995-4533-af6a-4347ccc0da5b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zk96k\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.594018 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.593996    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/289fb560-9995-4533-af6a-4347ccc0da5b-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zk96k\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.601982 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.601943    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzc84\" (UniqueName: \"kubernetes.io/projected/289fb560-9995-4533-af6a-4347ccc0da5b-kube-api-access-nzc84\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-zk96k\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.718343 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.718249    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:50:54.999177 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:54.999148    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"]
Sep 16 18:50:55.002005 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:50:55.001973    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289fb560_9995_4533_af6a_4347ccc0da5b.slice/crio-636623a1d9c91ab5c93a21d6e71e121bea745cefe25777af4f3a6d21be1f6eab WatchSource:0}: Error finding container 636623a1d9c91ab5c93a21d6e71e121bea745cefe25777af4f3a6d21be1f6eab: Status 404 returned error can't find the container with id 636623a1d9c91ab5c93a21d6e71e121bea745cefe25777af4f3a6d21be1f6eab
Sep 16 18:50:55.520171 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:55.520129    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" event={"ID":"289fb560-9995-4533-af6a-4347ccc0da5b","Type":"ContainerStarted","Data":"636623a1d9c91ab5c93a21d6e71e121bea745cefe25777af4f3a6d21be1f6eab"}
Sep 16 18:50:59.545921 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:50:59.545885    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" event={"ID":"289fb560-9995-4533-af6a-4347ccc0da5b","Type":"ContainerStarted","Data":"bfbe4d806048159b85a2da18f8a40990d180b8760468e6b525829a6790092e2a"}
Sep 16 18:51:03.565257 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:03.565218    2552 generic.go:356] "Generic (PLEG): container finished" podID="289fb560-9995-4533-af6a-4347ccc0da5b" containerID="bfbe4d806048159b85a2da18f8a40990d180b8760468e6b525829a6790092e2a" exitCode=0
Sep 16 18:51:03.565648 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:03.565297    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" event={"ID":"289fb560-9995-4533-af6a-4347ccc0da5b","Type":"ContainerDied","Data":"bfbe4d806048159b85a2da18f8a40990d180b8760468e6b525829a6790092e2a"}
Sep 16 18:51:24.016761 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:24.016702    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.27:8081/readyz\": dial tcp 10.132.0.27:8081: connect: connection refused"
Sep 16 18:51:25.698695 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:25.698661    2552 generic.go:356] "Generic (PLEG): container finished" podID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90" containerID="d68b3274b8fdf645dec2837b9a238739792e8cb06aee5f85ed72cb783bfbe247" exitCode=1
Sep 16 18:51:25.699248 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:25.698734    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerDied","Data":"d68b3274b8fdf645dec2837b9a238739792e8cb06aee5f85ed72cb783bfbe247"}
Sep 16 18:51:25.699248 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:25.698797    2552 scope.go:117] "RemoveContainer" containerID="bda2004f11373864e55c88d711297c357e1d880b408368ad6e8873a5c20f60a2"
Sep 16 18:51:25.699248 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:25.699102    2552 scope.go:117] "RemoveContainer" containerID="d68b3274b8fdf645dec2837b9a238739792e8cb06aee5f85ed72cb783bfbe247"
Sep 16 18:51:25.699434 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:51:25.699311    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 18:51:26.710071 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:26.710011    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" event={"ID":"289fb560-9995-4533-af6a-4347ccc0da5b","Type":"ContainerStarted","Data":"93cc1d9cb478bb0a05e6e92b9254e1566ee1c69be526f679a05ec7b6f1772029"}
Sep 16 18:51:26.787841 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:26.787790    2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:51:26.788304 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:26.788283    2552 scope.go:117] "RemoveContainer" containerID="d68b3274b8fdf645dec2837b9a238739792e8cb06aee5f85ed72cb783bfbe247"
Sep 16 18:51:26.788605 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:51:26.788561    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 18:51:29.016556 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:29.016512    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:51:29.016946 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:29.016916    2552 scope.go:117] "RemoveContainer" containerID="d68b3274b8fdf645dec2837b9a238739792e8cb06aee5f85ed72cb783bfbe247"
Sep 16 18:51:29.017130 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:51:29.017110    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 18:51:29.727826 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:29.727788    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" event={"ID":"289fb560-9995-4533-af6a-4347ccc0da5b","Type":"ContainerStarted","Data":"9a6a5927f0e819812429cd7d76ca92cca2e3bd77de16044b88477e4c5e9026e4"}
Sep 16 18:51:29.728037 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:29.727957    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:51:29.748098 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:29.748026    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" podStartSLOduration=2.875789043 podStartE2EDuration="35.748005233s" totalImagesPullingTime="28.707398914s" totalInitContainerRuntime="4.164817276s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:29.74611993 +0000 UTC m=+685.640307199" watchObservedRunningTime="2026-09-16 18:51:29.748005233 +0000 UTC m=+685.642192506"
Sep 16 18:51:30.733341 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:30.733298    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:51:30.734786 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:30.734748    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:51:31.737789 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:31.737732    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:51:36.742286 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:36.742253    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:51:36.742758 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:36.742732    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:51:43.619922 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:43.619881    2552 scope.go:117] "RemoveContainer" containerID="d68b3274b8fdf645dec2837b9a238739792e8cb06aee5f85ed72cb783bfbe247"
Sep 16 18:51:44.796722 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:44.796683    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerStarted","Data":"91f0f8a0764ded7e5266bad7363289d107583829977d6aca5fecb2637ae5594d"}
Sep 16 18:51:44.797270 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:44.796991    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:51:46.743649 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:46.743608    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:51:56.743278 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:51:56.743184    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:52:06.743617 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:06.743572    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:52:15.804128 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:15.804072    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:52:16.742710 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:16.742672    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:52:26.743495 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:26.743451    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:52:36.742857 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:36.742808    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:52:46.743269 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:46.743237    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:52:54.446455 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:54.446414    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"]
Sep 16 18:52:54.446867 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:54.446710    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kube-rbac-proxy" containerID="cri-o://9a6a5927f0e819812429cd7d76ca92cca2e3bd77de16044b88477e4c5e9026e4" gracePeriod=30
Sep 16 18:52:54.446867 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:54.446703    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kserve-container" containerID="cri-o://93cc1d9cb478bb0a05e6e92b9254e1566ee1c69be526f679a05ec7b6f1772029" gracePeriod=30
Sep 16 18:52:55.115399 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:55.115362    2552 generic.go:356] "Generic (PLEG): container finished" podID="289fb560-9995-4533-af6a-4347ccc0da5b" containerID="9a6a5927f0e819812429cd7d76ca92cca2e3bd77de16044b88477e4c5e9026e4" exitCode=2
Sep 16 18:52:55.115566 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:55.115436    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" event={"ID":"289fb560-9995-4533-af6a-4347ccc0da5b","Type":"ContainerDied","Data":"9a6a5927f0e819812429cd7d76ca92cca2e3bd77de16044b88477e4c5e9026e4"}
Sep 16 18:52:56.738980 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:56.738933    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 18:52:56.743292 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:56.743254    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:52:59.697813 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.697778    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:52:59.787896 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.787859    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/289fb560-9995-4533-af6a-4347ccc0da5b-proxy-tls\") pod \"289fb560-9995-4533-af6a-4347ccc0da5b\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") "
Sep 16 18:52:59.788043 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.787925    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/289fb560-9995-4533-af6a-4347ccc0da5b-kube-rbac-proxy-sar-config\") pod \"289fb560-9995-4533-af6a-4347ccc0da5b\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") "
Sep 16 18:52:59.788043 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.787970    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/289fb560-9995-4533-af6a-4347ccc0da5b-cabundle-cert\") pod \"289fb560-9995-4533-af6a-4347ccc0da5b\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") "
Sep 16 18:52:59.788043 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.787998    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzc84\" (UniqueName: \"kubernetes.io/projected/289fb560-9995-4533-af6a-4347ccc0da5b-kube-api-access-nzc84\") pod \"289fb560-9995-4533-af6a-4347ccc0da5b\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") "
Sep 16 18:52:59.788216 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.788050    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/289fb560-9995-4533-af6a-4347ccc0da5b-kserve-provision-location\") pod \"289fb560-9995-4533-af6a-4347ccc0da5b\" (UID: \"289fb560-9995-4533-af6a-4347ccc0da5b\") "
Sep 16 18:52:59.788404 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.788372    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289fb560-9995-4533-af6a-4347ccc0da5b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "289fb560-9995-4533-af6a-4347ccc0da5b" (UID: "289fb560-9995-4533-af6a-4347ccc0da5b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:59.788499 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.788380    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289fb560-9995-4533-af6a-4347ccc0da5b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "289fb560-9995-4533-af6a-4347ccc0da5b" (UID: "289fb560-9995-4533-af6a-4347ccc0da5b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:59.788499 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.788382    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289fb560-9995-4533-af6a-4347ccc0da5b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "289fb560-9995-4533-af6a-4347ccc0da5b" (UID: "289fb560-9995-4533-af6a-4347ccc0da5b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:59.790391 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.790333    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289fb560-9995-4533-af6a-4347ccc0da5b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "289fb560-9995-4533-af6a-4347ccc0da5b" (UID: "289fb560-9995-4533-af6a-4347ccc0da5b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:52:59.790504 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.790487    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289fb560-9995-4533-af6a-4347ccc0da5b-kube-api-access-nzc84" (OuterVolumeSpecName: "kube-api-access-nzc84") pod "289fb560-9995-4533-af6a-4347ccc0da5b" (UID: "289fb560-9995-4533-af6a-4347ccc0da5b"). InnerVolumeSpecName "kube-api-access-nzc84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:59.888955 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.888909    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/289fb560-9995-4533-af6a-4347ccc0da5b-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:52:59.888955 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.888947    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/289fb560-9995-4533-af6a-4347ccc0da5b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:52:59.888955 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.888956    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/289fb560-9995-4533-af6a-4347ccc0da5b-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:52:59.889230 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.888981    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzc84\" (UniqueName: \"kubernetes.io/projected/289fb560-9995-4533-af6a-4347ccc0da5b-kube-api-access-nzc84\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:52:59.889230 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:52:59.888991    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/289fb560-9995-4533-af6a-4347ccc0da5b-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:53:00.147121 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.147059    2552 generic.go:356] "Generic (PLEG): container finished" podID="289fb560-9995-4533-af6a-4347ccc0da5b" containerID="93cc1d9cb478bb0a05e6e92b9254e1566ee1c69be526f679a05ec7b6f1772029" exitCode=0
Sep 16 18:53:00.147309 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.147151    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"
Sep 16 18:53:00.147309 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.147145    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" event={"ID":"289fb560-9995-4533-af6a-4347ccc0da5b","Type":"ContainerDied","Data":"93cc1d9cb478bb0a05e6e92b9254e1566ee1c69be526f679a05ec7b6f1772029"}
Sep 16 18:53:00.147382 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.147309    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k" event={"ID":"289fb560-9995-4533-af6a-4347ccc0da5b","Type":"ContainerDied","Data":"636623a1d9c91ab5c93a21d6e71e121bea745cefe25777af4f3a6d21be1f6eab"}
Sep 16 18:53:00.147382 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.147338    2552 scope.go:117] "RemoveContainer" containerID="9a6a5927f0e819812429cd7d76ca92cca2e3bd77de16044b88477e4c5e9026e4"
Sep 16 18:53:00.156275 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.156249    2552 scope.go:117] "RemoveContainer" containerID="93cc1d9cb478bb0a05e6e92b9254e1566ee1c69be526f679a05ec7b6f1772029"
Sep 16 18:53:00.164936 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.164911    2552 scope.go:117] "RemoveContainer" containerID="bfbe4d806048159b85a2da18f8a40990d180b8760468e6b525829a6790092e2a"
Sep 16 18:53:00.173559 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.173519    2552 scope.go:117] "RemoveContainer" containerID="9a6a5927f0e819812429cd7d76ca92cca2e3bd77de16044b88477e4c5e9026e4"
Sep 16 18:53:00.174157 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:53:00.174119    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a6a5927f0e819812429cd7d76ca92cca2e3bd77de16044b88477e4c5e9026e4\": container with ID starting with 9a6a5927f0e819812429cd7d76ca92cca2e3bd77de16044b88477e4c5e9026e4 not found: ID does not exist" containerID="9a6a5927f0e819812429cd7d76ca92cca2e3bd77de16044b88477e4c5e9026e4"
Sep 16 18:53:00.174346 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.174311    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6a5927f0e819812429cd7d76ca92cca2e3bd77de16044b88477e4c5e9026e4"} err="failed to get container status \"9a6a5927f0e819812429cd7d76ca92cca2e3bd77de16044b88477e4c5e9026e4\": rpc error: code = NotFound desc = could not find container \"9a6a5927f0e819812429cd7d76ca92cca2e3bd77de16044b88477e4c5e9026e4\": container with ID starting with 9a6a5927f0e819812429cd7d76ca92cca2e3bd77de16044b88477e4c5e9026e4 not found: ID does not exist"
Sep 16 18:53:00.174459 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.174351    2552 scope.go:117] "RemoveContainer" containerID="93cc1d9cb478bb0a05e6e92b9254e1566ee1c69be526f679a05ec7b6f1772029"
Sep 16 18:53:00.174712 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:53:00.174692    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93cc1d9cb478bb0a05e6e92b9254e1566ee1c69be526f679a05ec7b6f1772029\": container with ID starting with 93cc1d9cb478bb0a05e6e92b9254e1566ee1c69be526f679a05ec7b6f1772029 not found: ID does not exist" containerID="93cc1d9cb478bb0a05e6e92b9254e1566ee1c69be526f679a05ec7b6f1772029"
Sep 16 18:53:00.174791 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.174720    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93cc1d9cb478bb0a05e6e92b9254e1566ee1c69be526f679a05ec7b6f1772029"} err="failed to get container status \"93cc1d9cb478bb0a05e6e92b9254e1566ee1c69be526f679a05ec7b6f1772029\": rpc error: code = NotFound desc = could not find container \"93cc1d9cb478bb0a05e6e92b9254e1566ee1c69be526f679a05ec7b6f1772029\": container with ID starting with 93cc1d9cb478bb0a05e6e92b9254e1566ee1c69be526f679a05ec7b6f1772029 not found: ID does not exist"
Sep 16 18:53:00.174791 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.174742    2552 scope.go:117] "RemoveContainer" containerID="bfbe4d806048159b85a2da18f8a40990d180b8760468e6b525829a6790092e2a"
Sep 16 18:53:00.175057 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:53:00.175038    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfbe4d806048159b85a2da18f8a40990d180b8760468e6b525829a6790092e2a\": container with ID starting with bfbe4d806048159b85a2da18f8a40990d180b8760468e6b525829a6790092e2a not found: ID does not exist" containerID="bfbe4d806048159b85a2da18f8a40990d180b8760468e6b525829a6790092e2a"
Sep 16 18:53:00.175123 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.175063    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfbe4d806048159b85a2da18f8a40990d180b8760468e6b525829a6790092e2a"} err="failed to get container status \"bfbe4d806048159b85a2da18f8a40990d180b8760468e6b525829a6790092e2a\": rpc error: code = NotFound desc = could not find container \"bfbe4d806048159b85a2da18f8a40990d180b8760468e6b525829a6790092e2a\": container with ID starting with bfbe4d806048159b85a2da18f8a40990d180b8760468e6b525829a6790092e2a not found: ID does not exist"
Sep 16 18:53:00.176292 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.176269    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"]
Sep 16 18:53:00.182308 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.182276    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k"]
Sep 16 18:53:00.625238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:53:00.625202    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" path="/var/lib/kubelet/pods/289fb560-9995-4533-af6a-4347ccc0da5b/volumes"
Sep 16 18:54:01.431233 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:01.431197    2552 generic.go:356] "Generic (PLEG): container finished" podID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90" containerID="91f0f8a0764ded7e5266bad7363289d107583829977d6aca5fecb2637ae5594d" exitCode=1
Sep 16 18:54:01.431753 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:01.431278    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerDied","Data":"91f0f8a0764ded7e5266bad7363289d107583829977d6aca5fecb2637ae5594d"}
Sep 16 18:54:01.431753 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:01.431361    2552 scope.go:117] "RemoveContainer" containerID="d68b3274b8fdf645dec2837b9a238739792e8cb06aee5f85ed72cb783bfbe247"
Sep 16 18:54:01.431753 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:01.431680    2552 scope.go:117] "RemoveContainer" containerID="91f0f8a0764ded7e5266bad7363289d107583829977d6aca5fecb2637ae5594d"
Sep 16 18:54:01.431951 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:54:01.431928    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 18:54:05.802420 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:05.802360    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:54:05.802835 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:05.802795    2552 scope.go:117] "RemoveContainer" containerID="91f0f8a0764ded7e5266bad7363289d107583829977d6aca5fecb2637ae5594d"
Sep 16 18:54:05.803030 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:54:05.803007    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 18:54:06.788583 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:06.788526    2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:54:06.788909 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:06.788895    2552 scope.go:117] "RemoveContainer" containerID="91f0f8a0764ded7e5266bad7363289d107583829977d6aca5fecb2637ae5594d"
Sep 16 18:54:06.789080 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:54:06.789060    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 18:54:09.554987 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.554947    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"]
Sep 16 18:54:09.555396 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.555287    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="storage-initializer"
Sep 16 18:54:09.555396 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.555299    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="storage-initializer"
Sep 16 18:54:09.555396 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.555310    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kube-rbac-proxy"
Sep 16 18:54:09.555396 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.555316    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kube-rbac-proxy"
Sep 16 18:54:09.555396 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.555336    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kserve-container"
Sep 16 18:54:09.555396 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.555342    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kserve-container"
Sep 16 18:54:09.555586 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.555405    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kube-rbac-proxy"
Sep 16 18:54:09.555586 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.555415    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="289fb560-9995-4533-af6a-4347ccc0da5b" containerName="kserve-container"
Sep 16 18:54:09.558199 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.558173    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:54:09.560747 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.560718    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 16 18:54:09.560945 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.560921    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 18:54:09.561026 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.560987    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:54:09.561105 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.560989    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 16 18:54:09.561783 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.561763    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:54:09.567541 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.567509    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"]
Sep 16 18:54:09.581630 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.581592    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97785da0-713a-44a1-9667-bfc42af412e2-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-c6scw\" (UID: \"97785da0-713a-44a1-9667-bfc42af412e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:54:09.581818 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.581644    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97785da0-713a-44a1-9667-bfc42af412e2-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-c6scw\" (UID: \"97785da0-713a-44a1-9667-bfc42af412e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:54:09.581818 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.581671    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9q8v\" (UniqueName: \"kubernetes.io/projected/97785da0-713a-44a1-9667-bfc42af412e2-kube-api-access-l9q8v\") pod \"message-dumper-predictor-6575bcfdfd-c6scw\" (UID: \"97785da0-713a-44a1-9667-bfc42af412e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:54:09.682891 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.682851    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97785da0-713a-44a1-9667-bfc42af412e2-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-c6scw\" (UID: \"97785da0-713a-44a1-9667-bfc42af412e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:54:09.683118 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.682904    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97785da0-713a-44a1-9667-bfc42af412e2-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-c6scw\" (UID: \"97785da0-713a-44a1-9667-bfc42af412e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:54:09.683118 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.682932    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9q8v\" (UniqueName: \"kubernetes.io/projected/97785da0-713a-44a1-9667-bfc42af412e2-kube-api-access-l9q8v\") pod \"message-dumper-predictor-6575bcfdfd-c6scw\" (UID: \"97785da0-713a-44a1-9667-bfc42af412e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:54:09.683606 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.683581    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97785da0-713a-44a1-9667-bfc42af412e2-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-c6scw\" (UID: \"97785da0-713a-44a1-9667-bfc42af412e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:54:09.685428 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.685406    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97785da0-713a-44a1-9667-bfc42af412e2-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-c6scw\" (UID: \"97785da0-713a-44a1-9667-bfc42af412e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:54:09.691683 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.691651    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9q8v\" (UniqueName: \"kubernetes.io/projected/97785da0-713a-44a1-9667-bfc42af412e2-kube-api-access-l9q8v\") pod \"message-dumper-predictor-6575bcfdfd-c6scw\" (UID: \"97785da0-713a-44a1-9667-bfc42af412e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:54:09.870282 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:09.870184    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:54:10.144474 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:10.144438    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"]
Sep 16 18:54:10.148606 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:54:10.148574    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97785da0_713a_44a1_9667_bfc42af412e2.slice/crio-f2eedbc14246a249e2edb733ec0c57d092866ce7b031175931f7b471580a0459 WatchSource:0}: Error finding container f2eedbc14246a249e2edb733ec0c57d092866ce7b031175931f7b471580a0459: Status 404 returned error can't find the container with id f2eedbc14246a249e2edb733ec0c57d092866ce7b031175931f7b471580a0459
Sep 16 18:54:10.474456 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:10.474368    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw" event={"ID":"97785da0-713a-44a1-9667-bfc42af412e2","Type":"ContainerStarted","Data":"f2eedbc14246a249e2edb733ec0c57d092866ce7b031175931f7b471580a0459"}
Sep 16 18:54:11.481124 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:11.481067    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw" event={"ID":"97785da0-713a-44a1-9667-bfc42af412e2","Type":"ContainerStarted","Data":"f0c150b38df69340ae63c15dbceb718e69d398c7788b41bce6fe9ee4740108b9"}
Sep 16 18:54:12.487474 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:12.487425    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw" event={"ID":"97785da0-713a-44a1-9667-bfc42af412e2","Type":"ContainerStarted","Data":"cbed3088d1bd4fdd43c88b1c55268674733a04ee863fb7d56b0893490a35fcc7"}
Sep 16 18:54:12.487934 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:12.487594    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:54:12.487934 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:12.487638    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:54:12.489448 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:12.489421    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:54:12.507033 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:12.506978    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw" podStartSLOduration=2.27978144 podStartE2EDuration="3.506961289s" totalImagesPullingTime="1.227179849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:12.5053987 +0000 UTC m=+848.399585984" watchObservedRunningTime="2026-09-16 18:54:12.506961289 +0000 UTC m=+848.401148556"
Sep 16 18:54:19.501061 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:19.501026    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:54:21.620479 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:21.620449    2552 scope.go:117] "RemoveContainer" containerID="91f0f8a0764ded7e5266bad7363289d107583829977d6aca5fecb2637ae5594d"
Sep 16 18:54:22.534277 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:22.534240    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerStarted","Data":"111f44595a72a6f133091c01eaf4e73b4172a6d60044b67a5c0806898a05ff59"}
Sep 16 18:54:22.534481 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:22.534460    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:54:53.540977 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:54:53.540942    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:56:04.605485 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:04.605444    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-c6scw_97785da0-713a-44a1-9667-bfc42af412e2/kserve-container/0.log"
Sep 16 18:56:04.788266 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:04.788225    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"]
Sep 16 18:56:04.788605 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:04.788550    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw" podUID="97785da0-713a-44a1-9667-bfc42af412e2" containerName="kserve-container" containerID="cri-o://f0c150b38df69340ae63c15dbceb718e69d398c7788b41bce6fe9ee4740108b9" gracePeriod=30
Sep 16 18:56:04.788605 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:04.788562    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw" podUID="97785da0-713a-44a1-9667-bfc42af412e2" containerName="kube-rbac-proxy" containerID="cri-o://cbed3088d1bd4fdd43c88b1c55268674733a04ee863fb7d56b0893490a35fcc7" gracePeriod=30
Sep 16 18:56:05.011820 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:05.011785    2552 generic.go:356] "Generic (PLEG): container finished" podID="97785da0-713a-44a1-9667-bfc42af412e2" containerID="cbed3088d1bd4fdd43c88b1c55268674733a04ee863fb7d56b0893490a35fcc7" exitCode=2
Sep 16 18:56:05.011820 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:05.011812    2552 generic.go:356] "Generic (PLEG): container finished" podID="97785da0-713a-44a1-9667-bfc42af412e2" containerID="f0c150b38df69340ae63c15dbceb718e69d398c7788b41bce6fe9ee4740108b9" exitCode=2
Sep 16 18:56:05.012041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:05.011861    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw" event={"ID":"97785da0-713a-44a1-9667-bfc42af412e2","Type":"ContainerDied","Data":"cbed3088d1bd4fdd43c88b1c55268674733a04ee863fb7d56b0893490a35fcc7"}
Sep 16 18:56:05.012041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:05.011920    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw" event={"ID":"97785da0-713a-44a1-9667-bfc42af412e2","Type":"ContainerDied","Data":"f0c150b38df69340ae63c15dbceb718e69d398c7788b41bce6fe9ee4740108b9"}
Sep 16 18:56:05.052564 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:05.052538    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:56:05.148692 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:05.148655    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9q8v\" (UniqueName: \"kubernetes.io/projected/97785da0-713a-44a1-9667-bfc42af412e2-kube-api-access-l9q8v\") pod \"97785da0-713a-44a1-9667-bfc42af412e2\" (UID: \"97785da0-713a-44a1-9667-bfc42af412e2\") "
Sep 16 18:56:05.148882 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:05.148752    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97785da0-713a-44a1-9667-bfc42af412e2-proxy-tls\") pod \"97785da0-713a-44a1-9667-bfc42af412e2\" (UID: \"97785da0-713a-44a1-9667-bfc42af412e2\") "
Sep 16 18:56:05.148882 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:05.148776    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97785da0-713a-44a1-9667-bfc42af412e2-kube-rbac-proxy-sar-config\") pod \"97785da0-713a-44a1-9667-bfc42af412e2\" (UID: \"97785da0-713a-44a1-9667-bfc42af412e2\") "
Sep 16 18:56:05.149236 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:05.149208    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97785da0-713a-44a1-9667-bfc42af412e2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "97785da0-713a-44a1-9667-bfc42af412e2" (UID: "97785da0-713a-44a1-9667-bfc42af412e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:05.150890 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:05.150821    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97785da0-713a-44a1-9667-bfc42af412e2-kube-api-access-l9q8v" (OuterVolumeSpecName: "kube-api-access-l9q8v") pod "97785da0-713a-44a1-9667-bfc42af412e2" (UID: "97785da0-713a-44a1-9667-bfc42af412e2"). InnerVolumeSpecName "kube-api-access-l9q8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:05.150890 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:05.150848    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97785da0-713a-44a1-9667-bfc42af412e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "97785da0-713a-44a1-9667-bfc42af412e2" (UID: "97785da0-713a-44a1-9667-bfc42af412e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:05.249956 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:05.249909    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9q8v\" (UniqueName: \"kubernetes.io/projected/97785da0-713a-44a1-9667-bfc42af412e2-kube-api-access-l9q8v\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:56:05.249956 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:05.249949    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97785da0-713a-44a1-9667-bfc42af412e2-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:56:05.249956 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:05.249965    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97785da0-713a-44a1-9667-bfc42af412e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:56:06.018303 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:06.018274    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"
Sep 16 18:56:06.018822 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:06.018313    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw" event={"ID":"97785da0-713a-44a1-9667-bfc42af412e2","Type":"ContainerDied","Data":"f2eedbc14246a249e2edb733ec0c57d092866ce7b031175931f7b471580a0459"}
Sep 16 18:56:06.018822 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:06.018396    2552 scope.go:117] "RemoveContainer" containerID="cbed3088d1bd4fdd43c88b1c55268674733a04ee863fb7d56b0893490a35fcc7"
Sep 16 18:56:06.027349 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:06.027323    2552 scope.go:117] "RemoveContainer" containerID="f0c150b38df69340ae63c15dbceb718e69d398c7788b41bce6fe9ee4740108b9"
Sep 16 18:56:06.040913 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:06.040876    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"]
Sep 16 18:56:06.049045 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:06.049007    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw"]
Sep 16 18:56:06.625590 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:06.625548    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97785da0-713a-44a1-9667-bfc42af412e2" path="/var/lib/kubelet/pods/97785da0-713a-44a1-9667-bfc42af412e2/volumes"
Sep 16 18:56:30.166312 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.166222    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"]
Sep 16 18:56:30.167039 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.166591    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97785da0-713a-44a1-9667-bfc42af412e2" containerName="kserve-container"
Sep 16 18:56:30.167039 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.166605    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="97785da0-713a-44a1-9667-bfc42af412e2" containerName="kserve-container"
Sep 16 18:56:30.167039 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.166615    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97785da0-713a-44a1-9667-bfc42af412e2" containerName="kube-rbac-proxy"
Sep 16 18:56:30.167039 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.166620    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="97785da0-713a-44a1-9667-bfc42af412e2" containerName="kube-rbac-proxy"
Sep 16 18:56:30.167039 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.166678    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="97785da0-713a-44a1-9667-bfc42af412e2" containerName="kserve-container"
Sep 16 18:56:30.167039 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.166688    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="97785da0-713a-44a1-9667-bfc42af412e2" containerName="kube-rbac-proxy"
Sep 16 18:56:30.171606 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.171573    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.173830 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.173802    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 16 18:56:30.174152 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.174121    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 16 18:56:30.174152 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.174137    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:56:30.174298 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.174146    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:56:30.174298 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.174241    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 18:56:30.174700 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.174682    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:56:30.174777 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.174708    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:56:30.180857 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.180828    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"]
Sep 16 18:56:30.278429 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.278389    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.278628 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.278441    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfd2r\" (UniqueName: \"kubernetes.io/projected/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kube-api-access-mfd2r\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.278628 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.278477    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.278628 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.278500    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.278744 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.278632    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.379166 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.379119    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.379166 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.379171    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.379499 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.379202    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfd2r\" (UniqueName: \"kubernetes.io/projected/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kube-api-access-mfd2r\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.379499 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.379227    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.379499 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.379249    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.379663 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.379585    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.379906 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.379883    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.380067 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.380045    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.381859 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.381834    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.388518 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.388494    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfd2r\" (UniqueName: \"kubernetes.io/projected/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kube-api-access-mfd2r\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.483978 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.483876    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:30.762229 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:30.762194    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"]
Sep 16 18:56:30.764842 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:56:30.764806    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311f61a6_7cf3_482a_a1d0_03bc97bbd85a.slice/crio-cf8df6e549dbf4dc5c5322a11c080bf2c9b623def3c40d8b02ab4fc8669afe52 WatchSource:0}: Error finding container cf8df6e549dbf4dc5c5322a11c080bf2c9b623def3c40d8b02ab4fc8669afe52: Status 404 returned error can't find the container with id cf8df6e549dbf4dc5c5322a11c080bf2c9b623def3c40d8b02ab4fc8669afe52
Sep 16 18:56:31.131780 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:31.131742    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" event={"ID":"311f61a6-7cf3-482a-a1d0-03bc97bbd85a","Type":"ContainerStarted","Data":"7f435327141bba20b8e6f55536794129ac0f8a113adaad6395efc0047f09cdd5"}
Sep 16 18:56:31.131986 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:31.131806    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" event={"ID":"311f61a6-7cf3-482a-a1d0-03bc97bbd85a","Type":"ContainerStarted","Data":"cf8df6e549dbf4dc5c5322a11c080bf2c9b623def3c40d8b02ab4fc8669afe52"}
Sep 16 18:56:35.152079 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:35.152036    2552 generic.go:356] "Generic (PLEG): container finished" podID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerID="7f435327141bba20b8e6f55536794129ac0f8a113adaad6395efc0047f09cdd5" exitCode=0
Sep 16 18:56:35.152540 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:35.152117    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" event={"ID":"311f61a6-7cf3-482a-a1d0-03bc97bbd85a","Type":"ContainerDied","Data":"7f435327141bba20b8e6f55536794129ac0f8a113adaad6395efc0047f09cdd5"}
Sep 16 18:56:36.159466 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:36.159425    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" event={"ID":"311f61a6-7cf3-482a-a1d0-03bc97bbd85a","Type":"ContainerStarted","Data":"a75104bff3f095ff89d719ebb0801156d99cb9d806506f80854e77828f03642b"}
Sep 16 18:56:36.159933 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:36.159496    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" event={"ID":"311f61a6-7cf3-482a-a1d0-03bc97bbd85a","Type":"ContainerStarted","Data":"521cd37eec2e1083b17ed46ac3ad5e46bd03852f06358d3addd2578e0cba776b"}
Sep 16 18:56:36.159933 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:36.159775    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:36.179941 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:36.179885    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" podStartSLOduration=2.391799437 podStartE2EDuration="6.179866082s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.788066645s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:36.17901584 +0000 UTC m=+992.073203110" watchObservedRunningTime="2026-09-16 18:56:36.179866082 +0000 UTC m=+992.074053350"
Sep 16 18:56:37.164014 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:37.163968    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:41.183415 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:41.183331    2552 generic.go:356] "Generic (PLEG): container finished" podID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90" containerID="111f44595a72a6f133091c01eaf4e73b4172a6d60044b67a5c0806898a05ff59" exitCode=1
Sep 16 18:56:41.183828 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:41.183404    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerDied","Data":"111f44595a72a6f133091c01eaf4e73b4172a6d60044b67a5c0806898a05ff59"}
Sep 16 18:56:41.183828 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:41.183472    2552 scope.go:117] "RemoveContainer" containerID="91f0f8a0764ded7e5266bad7363289d107583829977d6aca5fecb2637ae5594d"
Sep 16 18:56:41.183828 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:41.183772    2552 scope.go:117] "RemoveContainer" containerID="111f44595a72a6f133091c01eaf4e73b4172a6d60044b67a5c0806898a05ff59"
Sep 16 18:56:41.184020 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:56:41.183997    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 18:56:43.174397 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:43.174368    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:56:43.539590 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:43.539498    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:56:43.539882 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:43.539867    2552 scope.go:117] "RemoveContainer" containerID="111f44595a72a6f133091c01eaf4e73b4172a6d60044b67a5c0806898a05ff59"
Sep 16 18:56:43.540083 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:56:43.540064    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 18:56:46.788137 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:46.788054    2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:56:46.788538 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:46.788461    2552 scope.go:117] "RemoveContainer" containerID="111f44595a72a6f133091c01eaf4e73b4172a6d60044b67a5c0806898a05ff59"
Sep 16 18:56:46.788656 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:56:46.788637    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 18:56:59.620526 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:56:59.620484    2552 scope.go:117] "RemoveContainer" containerID="111f44595a72a6f133091c01eaf4e73b4172a6d60044b67a5c0806898a05ff59"
Sep 16 18:56:59.620979 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:56:59.620759    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 18:57:12.619694 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:57:12.619654    2552 scope.go:117] "RemoveContainer" containerID="111f44595a72a6f133091c01eaf4e73b4172a6d60044b67a5c0806898a05ff59"
Sep 16 18:57:12.620298 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:57:12.619883    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 18:57:13.175353 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:57:13.175308    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.35:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:57:23.175868 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:57:23.175813    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.35:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:57:27.619588 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:57:27.619548    2552 scope.go:117] "RemoveContainer" containerID="111f44595a72a6f133091c01eaf4e73b4172a6d60044b67a5c0806898a05ff59"
Sep 16 18:57:27.619969 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:57:27.619846    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:57:28.406002 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:57:28.405965    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerStarted","Data":"78ec68e33638101b4e6f01657100c5fe8c7fcc62ca7eac7347aa58d8f6f5e327"}
Sep 16 18:57:28.406288 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:57:28.406226    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:57:33.174992 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:57:33.174941    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.35:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:57:43.175082 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:57:43.175032    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.35:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:57:53.175808 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:57:53.175755    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.35:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:57:59.412587 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:57:59.412551    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:58:03.237848 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:03.237810    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:58:10.299304 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:10.299264    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"]
Sep 16 18:58:10.299781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:10.299542    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="kserve-container" containerID="cri-o://521cd37eec2e1083b17ed46ac3ad5e46bd03852f06358d3addd2578e0cba776b" gracePeriod=30
Sep 16 18:58:10.299781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:10.299579    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="kube-rbac-proxy" containerID="cri-o://a75104bff3f095ff89d719ebb0801156d99cb9d806506f80854e77828f03642b" gracePeriod=30
Sep 16 18:58:10.602526 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:10.602445    2552 generic.go:356] "Generic (PLEG): container finished" podID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerID="a75104bff3f095ff89d719ebb0801156d99cb9d806506f80854e77828f03642b" exitCode=2
Sep 16 18:58:10.602526 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:10.602491    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" event={"ID":"311f61a6-7cf3-482a-a1d0-03bc97bbd85a","Type":"ContainerDied","Data":"a75104bff3f095ff89d719ebb0801156d99cb9d806506f80854e77828f03642b"}
Sep 16 18:58:13.169386 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:13.169334    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 16 18:58:13.175967 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:13.175930    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.35:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:58:15.478969 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.478933    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"]
Sep 16 18:58:15.482546 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.482522    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:15.484631 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.484603    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 16 18:58:15.484788 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.484602    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 16 18:58:15.493368 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.493332    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"]
Sep 16 18:58:15.609653 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.609608    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4rrk\" (UniqueName: \"kubernetes.io/projected/e61ab900-58c8-4466-9ecb-48711f2f560b-kube-api-access-n4rrk\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:15.609823 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.609734    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e61ab900-58c8-4466-9ecb-48711f2f560b-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:15.609823 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.609788    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e61ab900-58c8-4466-9ecb-48711f2f560b-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:15.609823 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.609813    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e61ab900-58c8-4466-9ecb-48711f2f560b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:15.609931 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.609865    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e61ab900-58c8-4466-9ecb-48711f2f560b-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:15.627551 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.627511    2552 generic.go:356] "Generic (PLEG): container finished" podID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerID="521cd37eec2e1083b17ed46ac3ad5e46bd03852f06358d3addd2578e0cba776b" exitCode=0
Sep 16 18:58:15.627735 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.627584    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" event={"ID":"311f61a6-7cf3-482a-a1d0-03bc97bbd85a","Type":"ContainerDied","Data":"521cd37eec2e1083b17ed46ac3ad5e46bd03852f06358d3addd2578e0cba776b"}
Sep 16 18:58:15.661749 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.661712    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:58:15.710662 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.710550    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-proxy-tls\") pod \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") "
Sep 16 18:58:15.710662 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.710624    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kserve-provision-location\") pod \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") "
Sep 16 18:58:15.710861 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.710750    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kube-rbac-proxy-sar-config\") pod \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") "
Sep 16 18:58:15.710861 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.710805    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfd2r\" (UniqueName: \"kubernetes.io/projected/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kube-api-access-mfd2r\") pod \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") "
Sep 16 18:58:15.710861 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.710837    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-cabundle-cert\") pod \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\" (UID: \"311f61a6-7cf3-482a-a1d0-03bc97bbd85a\") "
Sep 16 18:58:15.711002 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.710952    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "311f61a6-7cf3-482a-a1d0-03bc97bbd85a" (UID: "311f61a6-7cf3-482a-a1d0-03bc97bbd85a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:15.711060 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.711040    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e61ab900-58c8-4466-9ecb-48711f2f560b-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:15.711155 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.711126    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e61ab900-58c8-4466-9ecb-48711f2f560b-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:15.711210 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.711156    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "311f61a6-7cf3-482a-a1d0-03bc97bbd85a" (UID: "311f61a6-7cf3-482a-a1d0-03bc97bbd85a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:15.711284 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:58:15.711207    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 16 18:58:15.711891 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:58:15.711347    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e61ab900-58c8-4466-9ecb-48711f2f560b-proxy-tls podName:e61ab900-58c8-4466-9ecb-48711f2f560b nodeName:}" failed. No retries permitted until 2026-09-16 18:58:16.211322489 +0000 UTC m=+1092.105509744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e61ab900-58c8-4466-9ecb-48711f2f560b-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" (UID: "e61ab900-58c8-4466-9ecb-48711f2f560b") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 16 18:58:15.711891 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.711168    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e61ab900-58c8-4466-9ecb-48711f2f560b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:15.711891 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.711396    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e61ab900-58c8-4466-9ecb-48711f2f560b-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:15.711891 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.711461    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4rrk\" (UniqueName: \"kubernetes.io/projected/e61ab900-58c8-4466-9ecb-48711f2f560b-kube-api-access-n4rrk\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:15.711891 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.711514    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:58:15.711891 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.711531    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:58:15.711891 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.711507    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "311f61a6-7cf3-482a-a1d0-03bc97bbd85a" (UID: "311f61a6-7cf3-482a-a1d0-03bc97bbd85a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:15.711891 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.711798    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e61ab900-58c8-4466-9ecb-48711f2f560b-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:15.711891 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.711807    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e61ab900-58c8-4466-9ecb-48711f2f560b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:15.711891 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.711893    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e61ab900-58c8-4466-9ecb-48711f2f560b-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:15.712983 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.712957    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "311f61a6-7cf3-482a-a1d0-03bc97bbd85a" (UID: "311f61a6-7cf3-482a-a1d0-03bc97bbd85a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:15.713613 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.713586    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kube-api-access-mfd2r" (OuterVolumeSpecName: "kube-api-access-mfd2r") pod "311f61a6-7cf3-482a-a1d0-03bc97bbd85a" (UID: "311f61a6-7cf3-482a-a1d0-03bc97bbd85a"). InnerVolumeSpecName "kube-api-access-mfd2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:15.722637 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.722603    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4rrk\" (UniqueName: \"kubernetes.io/projected/e61ab900-58c8-4466-9ecb-48711f2f560b-kube-api-access-n4rrk\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:15.812351 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.812306    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfd2r\" (UniqueName: \"kubernetes.io/projected/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-kube-api-access-mfd2r\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:58:15.812351 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.812339    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:58:15.812351 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:15.812351    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/311f61a6-7cf3-482a-a1d0-03bc97bbd85a-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:58:16.215715 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:16.215668    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e61ab900-58c8-4466-9ecb-48711f2f560b-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:16.218180 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:16.218151    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e61ab900-58c8-4466-9ecb-48711f2f560b-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:16.416629 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:16.416587    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:16.637044 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:16.637007    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf" event={"ID":"311f61a6-7cf3-482a-a1d0-03bc97bbd85a","Type":"ContainerDied","Data":"cf8df6e549dbf4dc5c5322a11c080bf2c9b623def3c40d8b02ab4fc8669afe52"}
Sep 16 18:58:16.637476 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:16.637045    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"
Sep 16 18:58:16.637476 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:16.637111    2552 scope.go:117] "RemoveContainer" containerID="a75104bff3f095ff89d719ebb0801156d99cb9d806506f80854e77828f03642b"
Sep 16 18:58:16.645753 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:16.645725    2552 scope.go:117] "RemoveContainer" containerID="521cd37eec2e1083b17ed46ac3ad5e46bd03852f06358d3addd2578e0cba776b"
Sep 16 18:58:16.654265 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:16.654241    2552 scope.go:117] "RemoveContainer" containerID="7f435327141bba20b8e6f55536794129ac0f8a113adaad6395efc0047f09cdd5"
Sep 16 18:58:16.656755 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:16.656725    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"]
Sep 16 18:58:16.661825 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:16.661790    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf"]
Sep 16 18:58:16.696754 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:16.696716    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"]
Sep 16 18:58:16.700004 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:58:16.699973    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61ab900_58c8_4466_9ecb_48711f2f560b.slice/crio-ca22572a4e47567a31f146e96f3f18c69f215d099703c64242c39bfdc6bd7cd5 WatchSource:0}: Error finding container ca22572a4e47567a31f146e96f3f18c69f215d099703c64242c39bfdc6bd7cd5: Status 404 returned error can't find the container with id ca22572a4e47567a31f146e96f3f18c69f215d099703c64242c39bfdc6bd7cd5
Sep 16 18:58:17.643848 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:17.643806    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" event={"ID":"e61ab900-58c8-4466-9ecb-48711f2f560b","Type":"ContainerStarted","Data":"436aee514f0828a70e64bd3f43a549695c871b40ce980ed32ad2f9cd2440f4fa"}
Sep 16 18:58:17.644361 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:17.643879    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" event={"ID":"e61ab900-58c8-4466-9ecb-48711f2f560b","Type":"ContainerStarted","Data":"ca22572a4e47567a31f146e96f3f18c69f215d099703c64242c39bfdc6bd7cd5"}
Sep 16 18:58:18.625798 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:18.625760    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" path="/var/lib/kubelet/pods/311f61a6-7cf3-482a-a1d0-03bc97bbd85a/volumes"
Sep 16 18:58:20.659317 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:20.659279    2552 generic.go:356] "Generic (PLEG): container finished" podID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerID="436aee514f0828a70e64bd3f43a549695c871b40ce980ed32ad2f9cd2440f4fa" exitCode=0
Sep 16 18:58:20.659699 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:20.659342    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" event={"ID":"e61ab900-58c8-4466-9ecb-48711f2f560b","Type":"ContainerDied","Data":"436aee514f0828a70e64bd3f43a549695c871b40ce980ed32ad2f9cd2440f4fa"}
Sep 16 18:58:21.666899 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:21.666862    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" event={"ID":"e61ab900-58c8-4466-9ecb-48711f2f560b","Type":"ContainerStarted","Data":"79df933e26dbde43ba4c4f32d81952d2c01f0bd3be982c2b14877bb968ec37a8"}
Sep 16 18:58:21.667330 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:21.666928    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" event={"ID":"e61ab900-58c8-4466-9ecb-48711f2f560b","Type":"ContainerStarted","Data":"1070bde49284766d5c46bcfe4b3178b5832751ddf54ae55d21be45ef3cbd1402"}
Sep 16 18:58:21.667330 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:21.667128    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:21.688700 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:21.688628    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" podStartSLOduration=2.850040187 podStartE2EDuration="6.688608567s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.83856838s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:21.687832152 +0000 UTC m=+1097.582019421" watchObservedRunningTime="2026-09-16 18:58:21.688608567 +0000 UTC m=+1097.582795836"
Sep 16 18:58:22.671506 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:22.671473    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:28.681774 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:28.681743    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:58:58.682842 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:58:58.682803    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 18:59:08.682726 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:08.682690    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 18:59:18.682259 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:18.682215    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 18:59:28.682468 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:28.682367    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 18:59:38.738966 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:38.738921    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:59:45.523220 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:45.523181    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"]
Sep 16 18:59:45.523691 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:45.523434    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="kserve-container" containerID="cri-o://1070bde49284766d5c46bcfe4b3178b5832751ddf54ae55d21be45ef3cbd1402" gracePeriod=30
Sep 16 18:59:45.523691 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:45.523477    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="kube-rbac-proxy" containerID="cri-o://79df933e26dbde43ba4c4f32d81952d2c01f0bd3be982c2b14877bb968ec37a8" gracePeriod=30
Sep 16 18:59:46.051522 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:46.051486    2552 generic.go:356] "Generic (PLEG): container finished" podID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerID="79df933e26dbde43ba4c4f32d81952d2c01f0bd3be982c2b14877bb968ec37a8" exitCode=2
Sep 16 18:59:46.051706 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:46.051558    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" event={"ID":"e61ab900-58c8-4466-9ecb-48711f2f560b","Type":"ContainerDied","Data":"79df933e26dbde43ba4c4f32d81952d2c01f0bd3be982c2b14877bb968ec37a8"}
Sep 16 18:59:46.788978 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:46.788930    2552 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.27:8081/healthz\": dial tcp 10.132.0.27:8081: connect: connection refused"
Sep 16 18:59:47.058624 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:47.058531    2552 generic.go:356] "Generic (PLEG): container finished" podID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90" containerID="78ec68e33638101b4e6f01657100c5fe8c7fcc62ca7eac7347aa58d8f6f5e327" exitCode=1
Sep 16 18:59:47.058624 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:47.058601    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerDied","Data":"78ec68e33638101b4e6f01657100c5fe8c7fcc62ca7eac7347aa58d8f6f5e327"}
Sep 16 18:59:47.058810 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:47.058665    2552 scope.go:117] "RemoveContainer" containerID="111f44595a72a6f133091c01eaf4e73b4172a6d60044b67a5c0806898a05ff59"
Sep 16 18:59:47.059013 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:47.058995    2552 scope.go:117] "RemoveContainer" containerID="78ec68e33638101b4e6f01657100c5fe8c7fcc62ca7eac7347aa58d8f6f5e327"
Sep 16 18:59:47.059220 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:59:47.059201    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 18:59:48.677141 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:48.677061    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 16 18:59:48.682567 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:48.682526    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 18:59:49.410624 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:49.410577    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:59:49.410989 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:49.410971    2552 scope.go:117] "RemoveContainer" containerID="78ec68e33638101b4e6f01657100c5fe8c7fcc62ca7eac7347aa58d8f6f5e327"
Sep 16 18:59:49.411202 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:59:49.411180    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 18:59:50.727391 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.727349    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"]
Sep 16 18:59:50.727908 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.727719    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="storage-initializer"
Sep 16 18:59:50.727908 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.727732    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="storage-initializer"
Sep 16 18:59:50.727908 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.727749    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="kube-rbac-proxy"
Sep 16 18:59:50.727908 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.727755    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="kube-rbac-proxy"
Sep 16 18:59:50.727908 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.727768    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="kserve-container"
Sep 16 18:59:50.727908 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.727775    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="kserve-container"
Sep 16 18:59:50.727908 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.727831    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="kube-rbac-proxy"
Sep 16 18:59:50.727908 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.727839    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="311f61a6-7cf3-482a-a1d0-03bc97bbd85a" containerName="kserve-container"
Sep 16 18:59:50.731061 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.731038    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.733324 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.733292    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 16 18:59:50.733459 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.733356    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 16 18:59:50.746145 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.746114    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"]
Sep 16 18:59:50.784513 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.784463    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d06411b3-0959-457b-8f5c-9f29b658e8b2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.784513 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.784508    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d06411b3-0959-457b-8f5c-9f29b658e8b2-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.784739 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.784597    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d06411b3-0959-457b-8f5c-9f29b658e8b2-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.784739 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.784649    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxb6b\" (UniqueName: \"kubernetes.io/projected/d06411b3-0959-457b-8f5c-9f29b658e8b2-kube-api-access-xxb6b\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.784813 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.784747    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d06411b3-0959-457b-8f5c-9f29b658e8b2-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.882573 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.882492    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:59:50.885426 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.885398    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4rrk\" (UniqueName: \"kubernetes.io/projected/e61ab900-58c8-4466-9ecb-48711f2f560b-kube-api-access-n4rrk\") pod \"e61ab900-58c8-4466-9ecb-48711f2f560b\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") "
Sep 16 18:59:50.885545 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.885444    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e61ab900-58c8-4466-9ecb-48711f2f560b-proxy-tls\") pod \"e61ab900-58c8-4466-9ecb-48711f2f560b\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") "
Sep 16 18:59:50.885545 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.885477    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e61ab900-58c8-4466-9ecb-48711f2f560b-kserve-provision-location\") pod \"e61ab900-58c8-4466-9ecb-48711f2f560b\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") "
Sep 16 18:59:50.885627 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.885585    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e61ab900-58c8-4466-9ecb-48711f2f560b-cabundle-cert\") pod \"e61ab900-58c8-4466-9ecb-48711f2f560b\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") "
Sep 16 18:59:50.885678 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.885637    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e61ab900-58c8-4466-9ecb-48711f2f560b-kube-rbac-proxy-sar-config\") pod \"e61ab900-58c8-4466-9ecb-48711f2f560b\" (UID: \"e61ab900-58c8-4466-9ecb-48711f2f560b\") "
Sep 16 18:59:50.885799 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.885775    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d06411b3-0959-457b-8f5c-9f29b658e8b2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.885888 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.885813    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d06411b3-0959-457b-8f5c-9f29b658e8b2-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.885888 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.885859    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61ab900-58c8-4466-9ecb-48711f2f560b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e61ab900-58c8-4466-9ecb-48711f2f560b" (UID: "e61ab900-58c8-4466-9ecb-48711f2f560b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:50.885994 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.885887    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d06411b3-0959-457b-8f5c-9f29b658e8b2-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.885994 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.885918    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxb6b\" (UniqueName: \"kubernetes.io/projected/d06411b3-0959-457b-8f5c-9f29b658e8b2-kube-api-access-xxb6b\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.885994 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.885977    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61ab900-58c8-4466-9ecb-48711f2f560b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e61ab900-58c8-4466-9ecb-48711f2f560b" (UID: "e61ab900-58c8-4466-9ecb-48711f2f560b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:50.886182 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.885993    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d06411b3-0959-457b-8f5c-9f29b658e8b2-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.886182 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.885982    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61ab900-58c8-4466-9ecb-48711f2f560b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e61ab900-58c8-4466-9ecb-48711f2f560b" (UID: "e61ab900-58c8-4466-9ecb-48711f2f560b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:50.886182 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.886139    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e61ab900-58c8-4466-9ecb-48711f2f560b-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:59:50.886182 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.886160    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e61ab900-58c8-4466-9ecb-48711f2f560b-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:59:50.886182 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.886176    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e61ab900-58c8-4466-9ecb-48711f2f560b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:59:50.886498 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.886469    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d06411b3-0959-457b-8f5c-9f29b658e8b2-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.886684 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.886660    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d06411b3-0959-457b-8f5c-9f29b658e8b2-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.886781 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.886660    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d06411b3-0959-457b-8f5c-9f29b658e8b2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.887920 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.887895    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e61ab900-58c8-4466-9ecb-48711f2f560b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e61ab900-58c8-4466-9ecb-48711f2f560b" (UID: "e61ab900-58c8-4466-9ecb-48711f2f560b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:50.888236 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.888216    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61ab900-58c8-4466-9ecb-48711f2f560b-kube-api-access-n4rrk" (OuterVolumeSpecName: "kube-api-access-n4rrk") pod "e61ab900-58c8-4466-9ecb-48711f2f560b" (UID: "e61ab900-58c8-4466-9ecb-48711f2f560b"). InnerVolumeSpecName "kube-api-access-n4rrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:50.888671 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.888650    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d06411b3-0959-457b-8f5c-9f29b658e8b2-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.896278 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.896250    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxb6b\" (UniqueName: \"kubernetes.io/projected/d06411b3-0959-457b-8f5c-9f29b658e8b2-kube-api-access-xxb6b\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:50.987213 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.987166    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4rrk\" (UniqueName: \"kubernetes.io/projected/e61ab900-58c8-4466-9ecb-48711f2f560b-kube-api-access-n4rrk\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:59:50.987213 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:50.987205    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e61ab900-58c8-4466-9ecb-48711f2f560b-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 18:59:51.042207 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.042168    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:51.083902 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.083866    2552 generic.go:356] "Generic (PLEG): container finished" podID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerID="1070bde49284766d5c46bcfe4b3178b5832751ddf54ae55d21be45ef3cbd1402" exitCode=0
Sep 16 18:59:51.084132 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.083950    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"
Sep 16 18:59:51.084132 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.083953    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" event={"ID":"e61ab900-58c8-4466-9ecb-48711f2f560b","Type":"ContainerDied","Data":"1070bde49284766d5c46bcfe4b3178b5832751ddf54ae55d21be45ef3cbd1402"}
Sep 16 18:59:51.084132 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.084104    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n" event={"ID":"e61ab900-58c8-4466-9ecb-48711f2f560b","Type":"ContainerDied","Data":"ca22572a4e47567a31f146e96f3f18c69f215d099703c64242c39bfdc6bd7cd5"}
Sep 16 18:59:51.084132 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.084124    2552 scope.go:117] "RemoveContainer" containerID="79df933e26dbde43ba4c4f32d81952d2c01f0bd3be982c2b14877bb968ec37a8"
Sep 16 18:59:51.094311 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.093770    2552 scope.go:117] "RemoveContainer" containerID="1070bde49284766d5c46bcfe4b3178b5832751ddf54ae55d21be45ef3cbd1402"
Sep 16 18:59:51.106041 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.106004    2552 scope.go:117] "RemoveContainer" containerID="436aee514f0828a70e64bd3f43a549695c871b40ce980ed32ad2f9cd2440f4fa"
Sep 16 18:59:51.115915 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.115895    2552 scope.go:117] "RemoveContainer" containerID="79df933e26dbde43ba4c4f32d81952d2c01f0bd3be982c2b14877bb968ec37a8"
Sep 16 18:59:51.116405 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.116381    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"]
Sep 16 18:59:51.116497 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:59:51.116380    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79df933e26dbde43ba4c4f32d81952d2c01f0bd3be982c2b14877bb968ec37a8\": container with ID starting with 79df933e26dbde43ba4c4f32d81952d2c01f0bd3be982c2b14877bb968ec37a8 not found: ID does not exist" containerID="79df933e26dbde43ba4c4f32d81952d2c01f0bd3be982c2b14877bb968ec37a8"
Sep 16 18:59:51.116497 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.116442    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79df933e26dbde43ba4c4f32d81952d2c01f0bd3be982c2b14877bb968ec37a8"} err="failed to get container status \"79df933e26dbde43ba4c4f32d81952d2c01f0bd3be982c2b14877bb968ec37a8\": rpc error: code = NotFound desc = could not find container \"79df933e26dbde43ba4c4f32d81952d2c01f0bd3be982c2b14877bb968ec37a8\": container with ID starting with 79df933e26dbde43ba4c4f32d81952d2c01f0bd3be982c2b14877bb968ec37a8 not found: ID does not exist"
Sep 16 18:59:51.116497 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.116465    2552 scope.go:117] "RemoveContainer" containerID="1070bde49284766d5c46bcfe4b3178b5832751ddf54ae55d21be45ef3cbd1402"
Sep 16 18:59:51.116783 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:59:51.116749    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1070bde49284766d5c46bcfe4b3178b5832751ddf54ae55d21be45ef3cbd1402\": container with ID starting with 1070bde49284766d5c46bcfe4b3178b5832751ddf54ae55d21be45ef3cbd1402 not found: ID does not exist" containerID="1070bde49284766d5c46bcfe4b3178b5832751ddf54ae55d21be45ef3cbd1402"
Sep 16 18:59:51.116892 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.116791    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1070bde49284766d5c46bcfe4b3178b5832751ddf54ae55d21be45ef3cbd1402"} err="failed to get container status \"1070bde49284766d5c46bcfe4b3178b5832751ddf54ae55d21be45ef3cbd1402\": rpc error: code = NotFound desc = could not find container \"1070bde49284766d5c46bcfe4b3178b5832751ddf54ae55d21be45ef3cbd1402\": container with ID starting with 1070bde49284766d5c46bcfe4b3178b5832751ddf54ae55d21be45ef3cbd1402 not found: ID does not exist"
Sep 16 18:59:51.116892 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.116814    2552 scope.go:117] "RemoveContainer" containerID="436aee514f0828a70e64bd3f43a549695c871b40ce980ed32ad2f9cd2440f4fa"
Sep 16 18:59:51.117137 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:59:51.117118    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"436aee514f0828a70e64bd3f43a549695c871b40ce980ed32ad2f9cd2440f4fa\": container with ID starting with 436aee514f0828a70e64bd3f43a549695c871b40ce980ed32ad2f9cd2440f4fa not found: ID does not exist" containerID="436aee514f0828a70e64bd3f43a549695c871b40ce980ed32ad2f9cd2440f4fa"
Sep 16 18:59:51.117204 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.117141    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"436aee514f0828a70e64bd3f43a549695c871b40ce980ed32ad2f9cd2440f4fa"} err="failed to get container status \"436aee514f0828a70e64bd3f43a549695c871b40ce980ed32ad2f9cd2440f4fa\": rpc error: code = NotFound desc = could not find container \"436aee514f0828a70e64bd3f43a549695c871b40ce980ed32ad2f9cd2440f4fa\": container with ID starting with 436aee514f0828a70e64bd3f43a549695c871b40ce980ed32ad2f9cd2440f4fa not found: ID does not exist"
Sep 16 18:59:51.138238 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.138083    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n"]
Sep 16 18:59:51.327792 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:51.327754    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"]
Sep 16 18:59:51.330860 ip-10-0-139-72 kubenswrapper[2552]: W0916 18:59:51.330818    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06411b3_0959_457b_8f5c_9f29b658e8b2.slice/crio-8aedf29b3f72f203a1ce85e51e03e899e7f04f08f2c2ff2e92e10b21f7f07f85 WatchSource:0}: Error finding container 8aedf29b3f72f203a1ce85e51e03e899e7f04f08f2c2ff2e92e10b21f7f07f85: Status 404 returned error can't find the container with id 8aedf29b3f72f203a1ce85e51e03e899e7f04f08f2c2ff2e92e10b21f7f07f85
Sep 16 18:59:52.090285 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:52.090244    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" event={"ID":"d06411b3-0959-457b-8f5c-9f29b658e8b2","Type":"ContainerStarted","Data":"1666217a6f198f40df71245371c6fed71570be6470dd396addb8eacecb4b8fd7"}
Sep 16 18:59:52.090728 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:52.090319    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" event={"ID":"d06411b3-0959-457b-8f5c-9f29b658e8b2","Type":"ContainerStarted","Data":"8aedf29b3f72f203a1ce85e51e03e899e7f04f08f2c2ff2e92e10b21f7f07f85"}
Sep 16 18:59:52.631983 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:52.631942    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" path="/var/lib/kubelet/pods/e61ab900-58c8-4466-9ecb-48711f2f560b/volumes"
Sep 16 18:59:56.112682 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:56.112647    2552 generic.go:356] "Generic (PLEG): container finished" podID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerID="1666217a6f198f40df71245371c6fed71570be6470dd396addb8eacecb4b8fd7" exitCode=0
Sep 16 18:59:56.113079 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:56.112721    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" event={"ID":"d06411b3-0959-457b-8f5c-9f29b658e8b2","Type":"ContainerDied","Data":"1666217a6f198f40df71245371c6fed71570be6470dd396addb8eacecb4b8fd7"}
Sep 16 18:59:56.788394 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:56.788347    2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 18:59:56.788776 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:56.788761    2552 scope.go:117] "RemoveContainer" containerID="78ec68e33638101b4e6f01657100c5fe8c7fcc62ca7eac7347aa58d8f6f5e327"
Sep 16 18:59:56.788958 ip-10-0-139-72 kubenswrapper[2552]: E0916 18:59:56.788940    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 18:59:57.119484 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:57.119393    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" event={"ID":"d06411b3-0959-457b-8f5c-9f29b658e8b2","Type":"ContainerStarted","Data":"a844665aa00cbfffc4bb4f82ad9c45aeadb36b023e5bf5ddaa3fb1cc2b8f73f8"}
Sep 16 18:59:57.119484 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:57.119463    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" event={"ID":"d06411b3-0959-457b-8f5c-9f29b658e8b2","Type":"ContainerStarted","Data":"f30f57e521cef91a6ec91b4b368082c6f83711a947c004926c1484a7e286745c"}
Sep 16 18:59:57.119896 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:57.119658    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 18:59:57.152414 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:57.152343    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" podStartSLOduration=3.071724024 podStartE2EDuration="7.152323723s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.080599699s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:57.14392521 +0000 UTC m=+1193.038112480" watchObservedRunningTime="2026-09-16 18:59:57.152323723 +0000 UTC m=+1193.046510993"
Sep 16 18:59:58.123720 ip-10-0-139-72 kubenswrapper[2552]: I0916 18:59:58.123681    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 19:00:04.135938 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:00:04.135906    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 19:00:11.619954 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:00:11.619913    2552 scope.go:117] "RemoveContainer" containerID="78ec68e33638101b4e6f01657100c5fe8c7fcc62ca7eac7347aa58d8f6f5e327"
Sep 16 19:00:11.620428 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:00:11.620172    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:00:22.620179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:00:22.620145    2552 scope.go:117] "RemoveContainer" containerID="78ec68e33638101b4e6f01657100c5fe8c7fcc62ca7eac7347aa58d8f6f5e327"
Sep 16 19:00:22.620635 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:00:22.620343    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:00:34.137817 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:00:34.137765    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:00:35.620414 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:00:35.620372    2552 scope.go:117] "RemoveContainer" containerID="78ec68e33638101b4e6f01657100c5fe8c7fcc62ca7eac7347aa58d8f6f5e327"
Sep 16 19:00:35.620799 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:00:35.620571    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:00:44.136576 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:00:44.136529    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:00:50.620397 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:00:50.620364    2552 scope.go:117] "RemoveContainer" containerID="78ec68e33638101b4e6f01657100c5fe8c7fcc62ca7eac7347aa58d8f6f5e327"
Sep 16 19:00:50.620807 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:00:50.620540    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:00:54.136684 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:00:54.136639    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:01:04.137154 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:04.137074    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:01:04.622412 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:04.622373    2552 scope.go:117] "RemoveContainer" containerID="78ec68e33638101b4e6f01657100c5fe8c7fcc62ca7eac7347aa58d8f6f5e327"
Sep 16 19:01:04.622609 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:01:04.622592    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:01:14.238146 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:14.238110    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 19:01:18.620390 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:18.620354    2552 scope.go:117] "RemoveContainer" containerID="78ec68e33638101b4e6f01657100c5fe8c7fcc62ca7eac7347aa58d8f6f5e327"
Sep 16 19:01:19.489050 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:19.489017    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerStarted","Data":"8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"}
Sep 16 19:01:19.489249 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:19.489232    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:01:20.849788 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:20.849747    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"]
Sep 16 19:01:20.850270 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:20.850008    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="kserve-container" containerID="cri-o://f30f57e521cef91a6ec91b4b368082c6f83711a947c004926c1484a7e286745c" gracePeriod=30
Sep 16 19:01:20.850270 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:20.850083    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="kube-rbac-proxy" containerID="cri-o://a844665aa00cbfffc4bb4f82ad9c45aeadb36b023e5bf5ddaa3fb1cc2b8f73f8" gracePeriod=30
Sep 16 19:01:21.500000 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:21.499964    2552 generic.go:356] "Generic (PLEG): container finished" podID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerID="a844665aa00cbfffc4bb4f82ad9c45aeadb36b023e5bf5ddaa3fb1cc2b8f73f8" exitCode=2
Sep 16 19:01:21.500208 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:21.500041    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" event={"ID":"d06411b3-0959-457b-8f5c-9f29b658e8b2","Type":"ContainerDied","Data":"a844665aa00cbfffc4bb4f82ad9c45aeadb36b023e5bf5ddaa3fb1cc2b8f73f8"}
Sep 16 19:01:24.129164 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:24.129118    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 16 19:01:24.137213 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:24.137177    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:01:26.505176 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.505141    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 19:01:26.526189 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.526114    2552 generic.go:356] "Generic (PLEG): container finished" podID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerID="f30f57e521cef91a6ec91b4b368082c6f83711a947c004926c1484a7e286745c" exitCode=0
Sep 16 19:01:26.526189 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.526155    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"
Sep 16 19:01:26.526594 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.526203    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" event={"ID":"d06411b3-0959-457b-8f5c-9f29b658e8b2","Type":"ContainerDied","Data":"f30f57e521cef91a6ec91b4b368082c6f83711a947c004926c1484a7e286745c"}
Sep 16 19:01:26.526594 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.526282    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565" event={"ID":"d06411b3-0959-457b-8f5c-9f29b658e8b2","Type":"ContainerDied","Data":"8aedf29b3f72f203a1ce85e51e03e899e7f04f08f2c2ff2e92e10b21f7f07f85"}
Sep 16 19:01:26.526594 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.526323    2552 scope.go:117] "RemoveContainer" containerID="a844665aa00cbfffc4bb4f82ad9c45aeadb36b023e5bf5ddaa3fb1cc2b8f73f8"
Sep 16 19:01:26.535684 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.535664    2552 scope.go:117] "RemoveContainer" containerID="f30f57e521cef91a6ec91b4b368082c6f83711a947c004926c1484a7e286745c"
Sep 16 19:01:26.544356 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.544328    2552 scope.go:117] "RemoveContainer" containerID="1666217a6f198f40df71245371c6fed71570be6470dd396addb8eacecb4b8fd7"
Sep 16 19:01:26.555613 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.555589    2552 scope.go:117] "RemoveContainer" containerID="a844665aa00cbfffc4bb4f82ad9c45aeadb36b023e5bf5ddaa3fb1cc2b8f73f8"
Sep 16 19:01:26.556001 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:01:26.555974    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a844665aa00cbfffc4bb4f82ad9c45aeadb36b023e5bf5ddaa3fb1cc2b8f73f8\": container with ID starting with a844665aa00cbfffc4bb4f82ad9c45aeadb36b023e5bf5ddaa3fb1cc2b8f73f8 not found: ID does not exist" containerID="a844665aa00cbfffc4bb4f82ad9c45aeadb36b023e5bf5ddaa3fb1cc2b8f73f8"
Sep 16 19:01:26.556126 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.556007    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a844665aa00cbfffc4bb4f82ad9c45aeadb36b023e5bf5ddaa3fb1cc2b8f73f8"} err="failed to get container status \"a844665aa00cbfffc4bb4f82ad9c45aeadb36b023e5bf5ddaa3fb1cc2b8f73f8\": rpc error: code = NotFound desc = could not find container \"a844665aa00cbfffc4bb4f82ad9c45aeadb36b023e5bf5ddaa3fb1cc2b8f73f8\": container with ID starting with a844665aa00cbfffc4bb4f82ad9c45aeadb36b023e5bf5ddaa3fb1cc2b8f73f8 not found: ID does not exist"
Sep 16 19:01:26.556126 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.556027    2552 scope.go:117] "RemoveContainer" containerID="f30f57e521cef91a6ec91b4b368082c6f83711a947c004926c1484a7e286745c"
Sep 16 19:01:26.556383 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:01:26.556360    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f30f57e521cef91a6ec91b4b368082c6f83711a947c004926c1484a7e286745c\": container with ID starting with f30f57e521cef91a6ec91b4b368082c6f83711a947c004926c1484a7e286745c not found: ID does not exist" containerID="f30f57e521cef91a6ec91b4b368082c6f83711a947c004926c1484a7e286745c"
Sep 16 19:01:26.556427 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.556392    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30f57e521cef91a6ec91b4b368082c6f83711a947c004926c1484a7e286745c"} err="failed to get container status \"f30f57e521cef91a6ec91b4b368082c6f83711a947c004926c1484a7e286745c\": rpc error: code = NotFound desc = could not find container \"f30f57e521cef91a6ec91b4b368082c6f83711a947c004926c1484a7e286745c\": container with ID starting with f30f57e521cef91a6ec91b4b368082c6f83711a947c004926c1484a7e286745c not found: ID does not exist"
Sep 16 19:01:26.556427 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.556410    2552 scope.go:117] "RemoveContainer" containerID="1666217a6f198f40df71245371c6fed71570be6470dd396addb8eacecb4b8fd7"
Sep 16 19:01:26.556691 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:01:26.556668    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1666217a6f198f40df71245371c6fed71570be6470dd396addb8eacecb4b8fd7\": container with ID starting with 1666217a6f198f40df71245371c6fed71570be6470dd396addb8eacecb4b8fd7 not found: ID does not exist" containerID="1666217a6f198f40df71245371c6fed71570be6470dd396addb8eacecb4b8fd7"
Sep 16 19:01:26.556787 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.556694    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1666217a6f198f40df71245371c6fed71570be6470dd396addb8eacecb4b8fd7"} err="failed to get container status \"1666217a6f198f40df71245371c6fed71570be6470dd396addb8eacecb4b8fd7\": rpc error: code = NotFound desc = could not find container \"1666217a6f198f40df71245371c6fed71570be6470dd396addb8eacecb4b8fd7\": container with ID starting with 1666217a6f198f40df71245371c6fed71570be6470dd396addb8eacecb4b8fd7 not found: ID does not exist"
Sep 16 19:01:26.604846 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.604806    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d06411b3-0959-457b-8f5c-9f29b658e8b2-proxy-tls\") pod \"d06411b3-0959-457b-8f5c-9f29b658e8b2\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") "
Sep 16 19:01:26.604846 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.604847    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d06411b3-0959-457b-8f5c-9f29b658e8b2-kube-rbac-proxy-sar-config\") pod \"d06411b3-0959-457b-8f5c-9f29b658e8b2\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") "
Sep 16 19:01:26.605126 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.604904    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d06411b3-0959-457b-8f5c-9f29b658e8b2-cabundle-cert\") pod \"d06411b3-0959-457b-8f5c-9f29b658e8b2\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") "
Sep 16 19:01:26.605126 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.604937    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxb6b\" (UniqueName: \"kubernetes.io/projected/d06411b3-0959-457b-8f5c-9f29b658e8b2-kube-api-access-xxb6b\") pod \"d06411b3-0959-457b-8f5c-9f29b658e8b2\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") "
Sep 16 19:01:26.605126 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.604959    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d06411b3-0959-457b-8f5c-9f29b658e8b2-kserve-provision-location\") pod \"d06411b3-0959-457b-8f5c-9f29b658e8b2\" (UID: \"d06411b3-0959-457b-8f5c-9f29b658e8b2\") "
Sep 16 19:01:26.605339 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.605311    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06411b3-0959-457b-8f5c-9f29b658e8b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d06411b3-0959-457b-8f5c-9f29b658e8b2" (UID: "d06411b3-0959-457b-8f5c-9f29b658e8b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:26.605403 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.605377    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06411b3-0959-457b-8f5c-9f29b658e8b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d06411b3-0959-457b-8f5c-9f29b658e8b2" (UID: "d06411b3-0959-457b-8f5c-9f29b658e8b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:26.605448 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.605394    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06411b3-0959-457b-8f5c-9f29b658e8b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d06411b3-0959-457b-8f5c-9f29b658e8b2" (UID: "d06411b3-0959-457b-8f5c-9f29b658e8b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:26.607254 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.607217    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06411b3-0959-457b-8f5c-9f29b658e8b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d06411b3-0959-457b-8f5c-9f29b658e8b2" (UID: "d06411b3-0959-457b-8f5c-9f29b658e8b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:26.607381 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.607318    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06411b3-0959-457b-8f5c-9f29b658e8b2-kube-api-access-xxb6b" (OuterVolumeSpecName: "kube-api-access-xxb6b") pod "d06411b3-0959-457b-8f5c-9f29b658e8b2" (UID: "d06411b3-0959-457b-8f5c-9f29b658e8b2"). InnerVolumeSpecName "kube-api-access-xxb6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:26.706243 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.706139    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d06411b3-0959-457b-8f5c-9f29b658e8b2-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:01:26.706243 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.706175    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d06411b3-0959-457b-8f5c-9f29b658e8b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:01:26.706243 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.706184    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d06411b3-0959-457b-8f5c-9f29b658e8b2-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:01:26.706243 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.706193    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxb6b\" (UniqueName: \"kubernetes.io/projected/d06411b3-0959-457b-8f5c-9f29b658e8b2-kube-api-access-xxb6b\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:01:26.706243 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.706202    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d06411b3-0959-457b-8f5c-9f29b658e8b2-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:01:26.843846 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.843809    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"]
Sep 16 19:01:26.845914 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:26.845885    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565"]
Sep 16 19:01:28.625483 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:28.625442    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" path="/var/lib/kubelet/pods/d06411b3-0959-457b-8f5c-9f29b658e8b2/volumes"
Sep 16 19:01:50.496137 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:01:50.496079    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:03:36.241502 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241462    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"]
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241781    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="kserve-container"
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241791    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="kserve-container"
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241810    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="kube-rbac-proxy"
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241815    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="kube-rbac-proxy"
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241826    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="storage-initializer"
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241832    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="storage-initializer"
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241840    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="kserve-container"
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241845    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="kserve-container"
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241852    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="storage-initializer"
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241857    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="storage-initializer"
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241868    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="kube-rbac-proxy"
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241874    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="kube-rbac-proxy"
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241920    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="kube-rbac-proxy"
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241929    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="kube-rbac-proxy"
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241934    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d06411b3-0959-457b-8f5c-9f29b658e8b2" containerName="kserve-container"
Sep 16 19:03:36.241974 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.241943    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e61ab900-58c8-4466-9ecb-48711f2f560b" containerName="kserve-container"
Sep 16 19:03:36.244921 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.244898    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.247185 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.247157    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 16 19:03:36.247357 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.247212    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:03:36.247357 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.247258    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:03:36.247357 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.247334    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 19:03:36.248012 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.247990    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:03:36.248145 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.248017    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:03:36.248145 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.248022    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:03:36.255476 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.255442    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"]
Sep 16 19:03:36.395785 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.395748    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d92ea23e-520e-432c-895e-aa4e84129c34-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.395986 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.395816    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d92ea23e-520e-432c-895e-aa4e84129c34-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.395986 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.395887    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d92ea23e-520e-432c-895e-aa4e84129c34-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.396147 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.395993    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvp2c\" (UniqueName: \"kubernetes.io/projected/d92ea23e-520e-432c-895e-aa4e84129c34-kube-api-access-cvp2c\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.396147 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.396042    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d92ea23e-520e-432c-895e-aa4e84129c34-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.497143 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.496994    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d92ea23e-520e-432c-895e-aa4e84129c34-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.497143 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.497055    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d92ea23e-520e-432c-895e-aa4e84129c34-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.497143 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.497123    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvp2c\" (UniqueName: \"kubernetes.io/projected/d92ea23e-520e-432c-895e-aa4e84129c34-kube-api-access-cvp2c\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.497553 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.497164    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d92ea23e-520e-432c-895e-aa4e84129c34-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.497553 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.497211    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d92ea23e-520e-432c-895e-aa4e84129c34-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.497553 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.497542    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d92ea23e-520e-432c-895e-aa4e84129c34-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.497849 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.497827    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d92ea23e-520e-432c-895e-aa4e84129c34-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.497898 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.497838    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d92ea23e-520e-432c-895e-aa4e84129c34-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.499705 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.499680    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d92ea23e-520e-432c-895e-aa4e84129c34-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.506637 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.506608    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvp2c\" (UniqueName: \"kubernetes.io/projected/d92ea23e-520e-432c-895e-aa4e84129c34-kube-api-access-cvp2c\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.557286 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.557237    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:03:36.833811 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:36.833775    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"]
Sep 16 19:03:36.836186 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:03:36.836152    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92ea23e_520e_432c_895e_aa4e84129c34.slice/crio-45436a380dcbcc1c1de0c0cf98cf90f494c438b3b04fdf4cd3c736986a600176 WatchSource:0}: Error finding container 45436a380dcbcc1c1de0c0cf98cf90f494c438b3b04fdf4cd3c736986a600176: Status 404 returned error can't find the container with id 45436a380dcbcc1c1de0c0cf98cf90f494c438b3b04fdf4cd3c736986a600176
Sep 16 19:03:37.121775 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:37.121739    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" event={"ID":"d92ea23e-520e-432c-895e-aa4e84129c34","Type":"ContainerStarted","Data":"c0f9432d462b47a71776609c328ccc63ccbf2171a64be17455b42874e1d3cb8a"}
Sep 16 19:03:37.121975 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:37.121807    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" event={"ID":"d92ea23e-520e-432c-895e-aa4e84129c34","Type":"ContainerStarted","Data":"45436a380dcbcc1c1de0c0cf98cf90f494c438b3b04fdf4cd3c736986a600176"}
Sep 16 19:03:39.132037 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:39.131987    2552 generic.go:356] "Generic (PLEG): container finished" podID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941" exitCode=1
Sep 16 19:03:39.132454 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:39.132052    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerDied","Data":"8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"}
Sep 16 19:03:39.132454 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:39.132136    2552 scope.go:117] "RemoveContainer" containerID="78ec68e33638101b4e6f01657100c5fe8c7fcc62ca7eac7347aa58d8f6f5e327"
Sep 16 19:03:39.132543 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:39.132530    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:03:39.132773 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:03:39.132748    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:03:40.494762 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:40.494727    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:03:40.495179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:40.495127    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:03:40.495344 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:03:40.495324    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:03:41.143361 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:41.143269    2552 generic.go:356] "Generic (PLEG): container finished" podID="d92ea23e-520e-432c-895e-aa4e84129c34" containerID="c0f9432d462b47a71776609c328ccc63ccbf2171a64be17455b42874e1d3cb8a" exitCode=0
Sep 16 19:03:41.143361 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:41.143344    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" event={"ID":"d92ea23e-520e-432c-895e-aa4e84129c34","Type":"ContainerDied","Data":"c0f9432d462b47a71776609c328ccc63ccbf2171a64be17455b42874e1d3cb8a"}
Sep 16 19:03:41.144684 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:41.144664    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:03:46.788293 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:46.788251    2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:03:46.788932 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:03:46.788782    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:03:46.789489 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:03:46.789011    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:04:00.620219 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:04:00.620185    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:04:00.620632 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:04:00.620394    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:04:12.619870 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:04:12.619828    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:04:12.620401 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:04:12.620056    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:04:24.622767 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:04:24.622728    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:04:24.623544 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:04:24.622952    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:04:38.620305 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:04:38.620260    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:04:38.620931 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:04:38.620500    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:04:50.620777 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:04:50.620741    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:04:50.621446 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:04:50.621014    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:05:02.620355 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:05:02.620308    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:05:02.620956 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:05:02.620540    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:05:16.620108 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:05:16.620054    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:05:16.620727 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:05:16.620278    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:05:30.619809 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:05:30.619722    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:05:30.620362 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:05:30.619992    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:05:43.620241 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:05:43.620203    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:05:43.620899 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:05:43.620506    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:05:56.620233 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:05:56.620198    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:05:56.620689 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:05:56.620388    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:05:57.863543 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:05:57.863489    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" event={"ID":"d92ea23e-520e-432c-895e-aa4e84129c34","Type":"ContainerStarted","Data":"79e4063844e131ec5d847f5a470b9d41ee77ff2c15a526db62b6f3d2efcb577f"}
Sep 16 19:05:57.864031 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:05:57.863575    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:05:57.864031 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:05:57.863658    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" event={"ID":"d92ea23e-520e-432c-895e-aa4e84129c34","Type":"ContainerStarted","Data":"1122b1ef46108265b22e50a0134076f3ccfe96375676461d40a4be243deeb46a"}
Sep 16 19:05:57.883958 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:05:57.883898    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" podStartSLOduration=2.144561517 podStartE2EDuration="2m21.883879187s" totalImagesPullingTime="2m15.756653085s" totalInitContainerRuntime="3.982664585s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:57.882585804 +0000 UTC m=+1553.776773087" watchObservedRunningTime="2026-09-16 19:05:57.883879187 +0000 UTC m=+1553.778066456"
Sep 16 19:05:58.867820 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:05:58.867779    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:06:01.489734 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.489696    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"]
Sep 16 19:06:01.739299 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.739254    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"]
Sep 16 19:06:01.739470 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.739387    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:01.742227 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.742145    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 16 19:06:01.742227 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.742165    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 16 19:06:01.767415 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.767367    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:01.767617 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.767437    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e00f1837-9a15-4eff-a14e-cade78dbe4ab-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:01.767617 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.767489    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e00f1837-9a15-4eff-a14e-cade78dbe4ab-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:01.767617 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.767562    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:01.767809 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.767628    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbms8\" (UniqueName: \"kubernetes.io/projected/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kube-api-access-rbms8\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:01.868646 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.868599    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbms8\" (UniqueName: \"kubernetes.io/projected/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kube-api-access-rbms8\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:01.868646 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.868648    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:01.868945 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.868672    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e00f1837-9a15-4eff-a14e-cade78dbe4ab-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:01.868945 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.868709    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e00f1837-9a15-4eff-a14e-cade78dbe4ab-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:01.868945 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.868732    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:01.868945 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:06:01.868838    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 16 19:06:01.868945 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:06:01.868905    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e00f1837-9a15-4eff-a14e-cade78dbe4ab-proxy-tls podName:e00f1837-9a15-4eff-a14e-cade78dbe4ab nodeName:}" failed. No retries permitted until 2026-09-16 19:06:02.368885599 +0000 UTC m=+1558.263072855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e00f1837-9a15-4eff-a14e-cade78dbe4ab-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" (UID: "e00f1837-9a15-4eff-a14e-cade78dbe4ab") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 16 19:06:01.869253 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.869132    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:01.869485 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.869467    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e00f1837-9a15-4eff-a14e-cade78dbe4ab-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:01.869523 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.869492    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:01.878878 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:01.878836    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbms8\" (UniqueName: \"kubernetes.io/projected/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kube-api-access-rbms8\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:02.374391 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:02.374347    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e00f1837-9a15-4eff-a14e-cade78dbe4ab-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:02.376947 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:02.376914    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e00f1837-9a15-4eff-a14e-cade78dbe4ab-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:02.650323 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:02.650222    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:06:03.051232 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:03.051204    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"]
Sep 16 19:06:03.054105 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:06:03.054047    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00f1837_9a15_4eff_a14e_cade78dbe4ab.slice/crio-bf587043dc57dced5df4e6cedff0545bdf8b97309967c6cc1a2b7cc497779f84 WatchSource:0}: Error finding container bf587043dc57dced5df4e6cedff0545bdf8b97309967c6cc1a2b7cc497779f84: Status 404 returned error can't find the container with id bf587043dc57dced5df4e6cedff0545bdf8b97309967c6cc1a2b7cc497779f84
Sep 16 19:06:03.900457 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:03.900260    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" event={"ID":"e00f1837-9a15-4eff-a14e-cade78dbe4ab","Type":"ContainerStarted","Data":"9322a514569e7c8657d2731cf7feddfbbda36296e663daa5e46244e36ec1c60a"}
Sep 16 19:06:03.900457 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:03.900343    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" event={"ID":"e00f1837-9a15-4eff-a14e-cade78dbe4ab","Type":"ContainerStarted","Data":"bf587043dc57dced5df4e6cedff0545bdf8b97309967c6cc1a2b7cc497779f84"}
Sep 16 19:06:04.878159 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:04.878128    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:06:08.626266 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:08.626229    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:06:08.626703 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:06:08.626456    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:06:08.928809 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:08.928717    2552 generic.go:356] "Generic (PLEG): container finished" podID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerID="9322a514569e7c8657d2731cf7feddfbbda36296e663daa5e46244e36ec1c60a" exitCode=0
Sep 16 19:06:08.928809 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:08.928792    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" event={"ID":"e00f1837-9a15-4eff-a14e-cade78dbe4ab","Type":"ContainerDied","Data":"9322a514569e7c8657d2731cf7feddfbbda36296e663daa5e46244e36ec1c60a"}
Sep 16 19:06:20.619916 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:20.619876    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:06:22.002455 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:22.002403    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerStarted","Data":"eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"}
Sep 16 19:06:22.002929 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:22.002662    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:06:34.933713 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:34.933660    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:06:44.931972 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:44.931914    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:06:53.010268 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:53.010231    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:06:54.881852 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:54.881811    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:06:56.365963 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:56.365732    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"]
Sep 16 19:06:56.366436 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:56.366226    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" containerName="kube-rbac-proxy" containerID="cri-o://79e4063844e131ec5d847f5a470b9d41ee77ff2c15a526db62b6f3d2efcb577f" gracePeriod=30
Sep 16 19:06:56.366511 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:56.366432    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" containerName="kserve-container" containerID="cri-o://1122b1ef46108265b22e50a0134076f3ccfe96375676461d40a4be243deeb46a" gracePeriod=30
Sep 16 19:06:57.188452 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:57.188411    2552 generic.go:356] "Generic (PLEG): container finished" podID="d92ea23e-520e-432c-895e-aa4e84129c34" containerID="79e4063844e131ec5d847f5a470b9d41ee77ff2c15a526db62b6f3d2efcb577f" exitCode=2
Sep 16 19:06:57.188662 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:57.188491    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" event={"ID":"d92ea23e-520e-432c-895e-aa4e84129c34","Type":"ContainerDied","Data":"79e4063844e131ec5d847f5a470b9d41ee77ff2c15a526db62b6f3d2efcb577f"}
Sep 16 19:06:59.872987 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:06:59.872933    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 16 19:07:04.873000 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:04.872949    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 16 19:07:04.879616 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:04.879573    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:07:05.673330 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.673295    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:07:05.777510 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.777412    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d92ea23e-520e-432c-895e-aa4e84129c34-cabundle-cert\") pod \"d92ea23e-520e-432c-895e-aa4e84129c34\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") "
Sep 16 19:07:05.777510 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.777473    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d92ea23e-520e-432c-895e-aa4e84129c34-proxy-tls\") pod \"d92ea23e-520e-432c-895e-aa4e84129c34\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") "
Sep 16 19:07:05.777754 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.777605    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d92ea23e-520e-432c-895e-aa4e84129c34-kserve-provision-location\") pod \"d92ea23e-520e-432c-895e-aa4e84129c34\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") "
Sep 16 19:07:05.777754 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.777657    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvp2c\" (UniqueName: \"kubernetes.io/projected/d92ea23e-520e-432c-895e-aa4e84129c34-kube-api-access-cvp2c\") pod \"d92ea23e-520e-432c-895e-aa4e84129c34\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") "
Sep 16 19:07:05.777754 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.777697    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d92ea23e-520e-432c-895e-aa4e84129c34-kube-rbac-proxy-sar-config\") pod \"d92ea23e-520e-432c-895e-aa4e84129c34\" (UID: \"d92ea23e-520e-432c-895e-aa4e84129c34\") "
Sep 16 19:07:05.777969 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.777946    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92ea23e-520e-432c-895e-aa4e84129c34-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d92ea23e-520e-432c-895e-aa4e84129c34" (UID: "d92ea23e-520e-432c-895e-aa4e84129c34"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:05.778051 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.777986    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92ea23e-520e-432c-895e-aa4e84129c34-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d92ea23e-520e-432c-895e-aa4e84129c34" (UID: "d92ea23e-520e-432c-895e-aa4e84129c34"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:05.778216 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.778191    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92ea23e-520e-432c-895e-aa4e84129c34-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d92ea23e-520e-432c-895e-aa4e84129c34" (UID: "d92ea23e-520e-432c-895e-aa4e84129c34"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:05.779822 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.779788    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92ea23e-520e-432c-895e-aa4e84129c34-kube-api-access-cvp2c" (OuterVolumeSpecName: "kube-api-access-cvp2c") pod "d92ea23e-520e-432c-895e-aa4e84129c34" (UID: "d92ea23e-520e-432c-895e-aa4e84129c34"). InnerVolumeSpecName "kube-api-access-cvp2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:05.779904 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.779857    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d92ea23e-520e-432c-895e-aa4e84129c34-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d92ea23e-520e-432c-895e-aa4e84129c34" (UID: "d92ea23e-520e-432c-895e-aa4e84129c34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:05.879156 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.879117    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d92ea23e-520e-432c-895e-aa4e84129c34-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:07:05.879156 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.879149    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d92ea23e-520e-432c-895e-aa4e84129c34-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:07:05.879156 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.879158    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d92ea23e-520e-432c-895e-aa4e84129c34-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:07:05.879156 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.879167    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvp2c\" (UniqueName: \"kubernetes.io/projected/d92ea23e-520e-432c-895e-aa4e84129c34-kube-api-access-cvp2c\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:07:05.879646 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:05.879178    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d92ea23e-520e-432c-895e-aa4e84129c34-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:07:06.240941 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.240898    2552 generic.go:356] "Generic (PLEG): container finished" podID="d92ea23e-520e-432c-895e-aa4e84129c34" containerID="1122b1ef46108265b22e50a0134076f3ccfe96375676461d40a4be243deeb46a" exitCode=0
Sep 16 19:07:06.241164 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.240980    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"
Sep 16 19:07:06.241164 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.240987    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" event={"ID":"d92ea23e-520e-432c-895e-aa4e84129c34","Type":"ContainerDied","Data":"1122b1ef46108265b22e50a0134076f3ccfe96375676461d40a4be243deeb46a"}
Sep 16 19:07:06.241164 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.241048    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl" event={"ID":"d92ea23e-520e-432c-895e-aa4e84129c34","Type":"ContainerDied","Data":"45436a380dcbcc1c1de0c0cf98cf90f494c438b3b04fdf4cd3c736986a600176"}
Sep 16 19:07:06.241164 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.241107    2552 scope.go:117] "RemoveContainer" containerID="79e4063844e131ec5d847f5a470b9d41ee77ff2c15a526db62b6f3d2efcb577f"
Sep 16 19:07:06.252349 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.252273    2552 scope.go:117] "RemoveContainer" containerID="1122b1ef46108265b22e50a0134076f3ccfe96375676461d40a4be243deeb46a"
Sep 16 19:07:06.261389 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.261359    2552 scope.go:117] "RemoveContainer" containerID="c0f9432d462b47a71776609c328ccc63ccbf2171a64be17455b42874e1d3cb8a"
Sep 16 19:07:06.277106 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.269575    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"]
Sep 16 19:07:06.277106 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.274642    2552 scope.go:117] "RemoveContainer" containerID="79e4063844e131ec5d847f5a470b9d41ee77ff2c15a526db62b6f3d2efcb577f"
Sep 16 19:07:06.277106 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.274685    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl"]
Sep 16 19:07:06.277106 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:07:06.276129    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e4063844e131ec5d847f5a470b9d41ee77ff2c15a526db62b6f3d2efcb577f\": container with ID starting with 79e4063844e131ec5d847f5a470b9d41ee77ff2c15a526db62b6f3d2efcb577f not found: ID does not exist" containerID="79e4063844e131ec5d847f5a470b9d41ee77ff2c15a526db62b6f3d2efcb577f"
Sep 16 19:07:06.277106 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.276172    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e4063844e131ec5d847f5a470b9d41ee77ff2c15a526db62b6f3d2efcb577f"} err="failed to get container status \"79e4063844e131ec5d847f5a470b9d41ee77ff2c15a526db62b6f3d2efcb577f\": rpc error: code = NotFound desc = could not find container \"79e4063844e131ec5d847f5a470b9d41ee77ff2c15a526db62b6f3d2efcb577f\": container with ID starting with 79e4063844e131ec5d847f5a470b9d41ee77ff2c15a526db62b6f3d2efcb577f not found: ID does not exist"
Sep 16 19:07:06.277106 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.276201    2552 scope.go:117] "RemoveContainer" containerID="1122b1ef46108265b22e50a0134076f3ccfe96375676461d40a4be243deeb46a"
Sep 16 19:07:06.277826 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:07:06.277789    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1122b1ef46108265b22e50a0134076f3ccfe96375676461d40a4be243deeb46a\": container with ID starting with 1122b1ef46108265b22e50a0134076f3ccfe96375676461d40a4be243deeb46a not found: ID does not exist" containerID="1122b1ef46108265b22e50a0134076f3ccfe96375676461d40a4be243deeb46a"
Sep 16 19:07:06.277965 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.277833    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1122b1ef46108265b22e50a0134076f3ccfe96375676461d40a4be243deeb46a"} err="failed to get container status \"1122b1ef46108265b22e50a0134076f3ccfe96375676461d40a4be243deeb46a\": rpc error: code = NotFound desc = could not find container \"1122b1ef46108265b22e50a0134076f3ccfe96375676461d40a4be243deeb46a\": container with ID starting with 1122b1ef46108265b22e50a0134076f3ccfe96375676461d40a4be243deeb46a not found: ID does not exist"
Sep 16 19:07:06.278052 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.277860    2552 scope.go:117] "RemoveContainer" containerID="c0f9432d462b47a71776609c328ccc63ccbf2171a64be17455b42874e1d3cb8a"
Sep 16 19:07:06.278548 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:07:06.278516    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f9432d462b47a71776609c328ccc63ccbf2171a64be17455b42874e1d3cb8a\": container with ID starting with c0f9432d462b47a71776609c328ccc63ccbf2171a64be17455b42874e1d3cb8a not found: ID does not exist" containerID="c0f9432d462b47a71776609c328ccc63ccbf2171a64be17455b42874e1d3cb8a"
Sep 16 19:07:06.278683 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.278545    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f9432d462b47a71776609c328ccc63ccbf2171a64be17455b42874e1d3cb8a"} err="failed to get container status \"c0f9432d462b47a71776609c328ccc63ccbf2171a64be17455b42874e1d3cb8a\": rpc error: code = NotFound desc = could not find container \"c0f9432d462b47a71776609c328ccc63ccbf2171a64be17455b42874e1d3cb8a\": container with ID starting with c0f9432d462b47a71776609c328ccc63ccbf2171a64be17455b42874e1d3cb8a not found: ID does not exist"
Sep 16 19:07:06.627055 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:06.627014    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" path="/var/lib/kubelet/pods/d92ea23e-520e-432c-895e-aa4e84129c34/volumes"
Sep 16 19:07:38.617958 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.617919    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"]
Sep 16 19:07:38.618617 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.618443    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" containerName="kube-rbac-proxy"
Sep 16 19:07:38.618617 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.618461    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" containerName="kube-rbac-proxy"
Sep 16 19:07:38.618617 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.618476    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" containerName="kserve-container"
Sep 16 19:07:38.618617 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.618486    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" containerName="kserve-container"
Sep 16 19:07:38.618617 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.618509    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" containerName="storage-initializer"
Sep 16 19:07:38.618617 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.618519    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" containerName="storage-initializer"
Sep 16 19:07:38.618617 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.618607    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" containerName="kube-rbac-proxy"
Sep 16 19:07:38.618617 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.618622    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d92ea23e-520e-432c-895e-aa4e84129c34" containerName="kserve-container"
Sep 16 19:07:38.750498 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.750414    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:38.752813 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.752778    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 16 19:07:38.752813 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.752780    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:07:38.758298 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.758263    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"]
Sep 16 19:07:38.874504 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.874466    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/106aa30b-43e8-4356-94fa-0bfa18dfaad5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:38.874701 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.874528    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhtn8\" (UniqueName: \"kubernetes.io/projected/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kube-api-access-dhtn8\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:38.874701 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.874626    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/106aa30b-43e8-4356-94fa-0bfa18dfaad5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:38.874701 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.874683    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:38.874859 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.874736    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:38.975724 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.975680    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/106aa30b-43e8-4356-94fa-0bfa18dfaad5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:38.975920 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.975744    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhtn8\" (UniqueName: \"kubernetes.io/projected/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kube-api-access-dhtn8\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:38.975920 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.975792    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/106aa30b-43e8-4356-94fa-0bfa18dfaad5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:38.975920 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.975831    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:38.975920 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.975858    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:38.976823 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.976794    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:38.976942 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.976830    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/106aa30b-43e8-4356-94fa-0bfa18dfaad5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:38.977104 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.977058    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:38.979372 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.979341    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/106aa30b-43e8-4356-94fa-0bfa18dfaad5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:38.985681 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:38.985650    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhtn8\" (UniqueName: \"kubernetes.io/projected/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kube-api-access-dhtn8\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:07:39.064555 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:07:39.064511    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:08:10.952559 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:10.952520    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"]
Sep 16 19:08:10.955174 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:08:10.955139    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106aa30b_43e8_4356_94fa_0bfa18dfaad5.slice/crio-260a900fbe55791df6239c988bf4beed5babaa1a250a53b73e078d06ffb1b900 WatchSource:0}: Error finding container 260a900fbe55791df6239c988bf4beed5babaa1a250a53b73e078d06ffb1b900: Status 404 returned error can't find the container with id 260a900fbe55791df6239c988bf4beed5babaa1a250a53b73e078d06ffb1b900
Sep 16 19:08:11.603467 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:11.603353    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" event={"ID":"e00f1837-9a15-4eff-a14e-cade78dbe4ab","Type":"ContainerStarted","Data":"49df157e0a16179cb62fdbad70a3d7d217d23ca760ef868e8b688bffe79c7928"}
Sep 16 19:08:11.603467 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:11.603445    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:08:11.603731 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:11.603478    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" event={"ID":"e00f1837-9a15-4eff-a14e-cade78dbe4ab","Type":"ContainerStarted","Data":"b478c6478f321c9fb85b999a48d0dcea5ec102aaca4dba6e51d6dce9192274dd"}
Sep 16 19:08:11.603731 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:11.603510    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:08:11.605335 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:11.605302    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h" event={"ID":"106aa30b-43e8-4356-94fa-0bfa18dfaad5","Type":"ContainerStarted","Data":"e94d0db3c3dfb13ff46a1c9c7211952c12534168da04d27af26a253ab42175aa"}
Sep 16 19:08:11.605473 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:11.605378    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h" event={"ID":"106aa30b-43e8-4356-94fa-0bfa18dfaad5","Type":"ContainerStarted","Data":"260a900fbe55791df6239c988bf4beed5babaa1a250a53b73e078d06ffb1b900"}
Sep 16 19:08:11.605658 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:11.605483    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:08:11.625354 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:11.625296    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" podStartSLOduration=3.576536578 podStartE2EDuration="2m10.625277957s" totalImagesPullingTime="2m2.283902658s" totalInitContainerRuntime="4.764838721s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:11.624103624 +0000 UTC m=+1687.518290889" watchObservedRunningTime="2026-09-16 19:08:11.625277957 +0000 UTC m=+1687.519465226"
Sep 16 19:08:12.610736 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:12.610685    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:08:15.625365 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:15.625326    2552 generic.go:356] "Generic (PLEG): container finished" podID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerID="e94d0db3c3dfb13ff46a1c9c7211952c12534168da04d27af26a253ab42175aa" exitCode=0
Sep 16 19:08:15.625757 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:15.625397    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h" event={"ID":"106aa30b-43e8-4356-94fa-0bfa18dfaad5","Type":"ContainerDied","Data":"e94d0db3c3dfb13ff46a1c9c7211952c12534168da04d27af26a253ab42175aa"}
Sep 16 19:08:16.632540 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:16.632499    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h" event={"ID":"106aa30b-43e8-4356-94fa-0bfa18dfaad5","Type":"ContainerStarted","Data":"a151d46a1a329a42aafb7628166aeea3fb24ec4ef6ee18daac65c89fb5bd6059"}
Sep 16 19:08:16.633034 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:16.632563    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h" event={"ID":"106aa30b-43e8-4356-94fa-0bfa18dfaad5","Type":"ContainerStarted","Data":"e4a9a858abd53eb4a526ebfb4801721ed0581679692bd1b9588f30cfa3fbb4e6"}
Sep 16 19:08:16.633034 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:16.632905    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:08:16.633034 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:16.632964    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:08:16.659225 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:16.659165    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h" podStartSLOduration=34.500797116 podStartE2EDuration="38.659145636s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.15834852s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:16.65831411 +0000 UTC m=+1692.552501408" watchObservedRunningTime="2026-09-16 19:08:16.659145636 +0000 UTC m=+1692.553332901"
Sep 16 19:08:17.615779 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:17.615744    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:08:17.616378 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:17.616348    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:08:22.645447 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:22.645409    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:08:27.617316 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:27.617232    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:08:40.753738 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:40.753701    2552 generic.go:356] "Generic (PLEG): container finished" podID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc" exitCode=1
Sep 16 19:08:40.754314 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:40.753780    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerDied","Data":"eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"}
Sep 16 19:08:40.754314 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:40.753843    2552 scope.go:117] "RemoveContainer" containerID="8707a35cf23b7d9964916f8f6887d4ed7216b8640c58e96a5c68a0ef3ec06941"
Sep 16 19:08:40.754314 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:40.754243    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:08:40.754508 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:08:40.754477    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:08:43.008724 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:43.008679    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:08:43.009156 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:43.009071    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:08:43.009303 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:08:43.009274    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:08:45.390262 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:45.390221    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"]
Sep 16 19:08:45.390673 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:45.390498    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerName="kserve-container" containerID="cri-o://b478c6478f321c9fb85b999a48d0dcea5ec102aaca4dba6e51d6dce9192274dd" gracePeriod=30
Sep 16 19:08:45.390673 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:45.390572    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerName="kube-rbac-proxy" containerID="cri-o://49df157e0a16179cb62fdbad70a3d7d217d23ca760ef868e8b688bffe79c7928" gracePeriod=30
Sep 16 19:08:45.783323 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:45.783225    2552 generic.go:356] "Generic (PLEG): container finished" podID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerID="49df157e0a16179cb62fdbad70a3d7d217d23ca760ef868e8b688bffe79c7928" exitCode=2
Sep 16 19:08:45.783323 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:45.783289    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" event={"ID":"e00f1837-9a15-4eff-a14e-cade78dbe4ab","Type":"ContainerDied","Data":"49df157e0a16179cb62fdbad70a3d7d217d23ca760ef868e8b688bffe79c7928"}
Sep 16 19:08:46.787916 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:46.787884    2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:08:46.788340 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:46.788273    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:08:46.788474 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:08:46.788452    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:08:47.611363 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:47.611265    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 16 19:08:47.616295 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:47.616264    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:08:47.794779 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:47.794737    2552 generic.go:356] "Generic (PLEG): container finished" podID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerID="b478c6478f321c9fb85b999a48d0dcea5ec102aaca4dba6e51d6dce9192274dd" exitCode=0
Sep 16 19:08:47.795203 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:47.794809    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" event={"ID":"e00f1837-9a15-4eff-a14e-cade78dbe4ab","Type":"ContainerDied","Data":"b478c6478f321c9fb85b999a48d0dcea5ec102aaca4dba6e51d6dce9192274dd"}
Sep 16 19:08:47.994797 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:47.994760    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:08:48.037913 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.037857    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kube-rbac-proxy-sar-config\") pod \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") "
Sep 16 19:08:48.037913 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.037921    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e00f1837-9a15-4eff-a14e-cade78dbe4ab-cabundle-cert\") pod \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") "
Sep 16 19:08:48.038244 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.037952    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbms8\" (UniqueName: \"kubernetes.io/projected/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kube-api-access-rbms8\") pod \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") "
Sep 16 19:08:48.038244 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.038041    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kserve-provision-location\") pod \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") "
Sep 16 19:08:48.038244 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.038120    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e00f1837-9a15-4eff-a14e-cade78dbe4ab-proxy-tls\") pod \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\" (UID: \"e00f1837-9a15-4eff-a14e-cade78dbe4ab\") "
Sep 16 19:08:48.038643 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.038604    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e00f1837-9a15-4eff-a14e-cade78dbe4ab" (UID: "e00f1837-9a15-4eff-a14e-cade78dbe4ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:48.038764 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.038678    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00f1837-9a15-4eff-a14e-cade78dbe4ab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e00f1837-9a15-4eff-a14e-cade78dbe4ab" (UID: "e00f1837-9a15-4eff-a14e-cade78dbe4ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:48.039067 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.039045    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e00f1837-9a15-4eff-a14e-cade78dbe4ab" (UID: "e00f1837-9a15-4eff-a14e-cade78dbe4ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:48.040411 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.040382    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e00f1837-9a15-4eff-a14e-cade78dbe4ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e00f1837-9a15-4eff-a14e-cade78dbe4ab" (UID: "e00f1837-9a15-4eff-a14e-cade78dbe4ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:48.040763 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.040739    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kube-api-access-rbms8" (OuterVolumeSpecName: "kube-api-access-rbms8") pod "e00f1837-9a15-4eff-a14e-cade78dbe4ab" (UID: "e00f1837-9a15-4eff-a14e-cade78dbe4ab"). InnerVolumeSpecName "kube-api-access-rbms8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:48.139574 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.139466    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:08:48.139574 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.139503    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e00f1837-9a15-4eff-a14e-cade78dbe4ab-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:08:48.139574 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.139512    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbms8\" (UniqueName: \"kubernetes.io/projected/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kube-api-access-rbms8\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:08:48.139574 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.139521    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e00f1837-9a15-4eff-a14e-cade78dbe4ab-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:08:48.139574 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.139530    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e00f1837-9a15-4eff-a14e-cade78dbe4ab-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:08:48.802284 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.802242    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp" event={"ID":"e00f1837-9a15-4eff-a14e-cade78dbe4ab","Type":"ContainerDied","Data":"bf587043dc57dced5df4e6cedff0545bdf8b97309967c6cc1a2b7cc497779f84"}
Sep 16 19:08:48.802743 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.802316    2552 scope.go:117] "RemoveContainer" containerID="49df157e0a16179cb62fdbad70a3d7d217d23ca760ef868e8b688bffe79c7928"
Sep 16 19:08:48.802743 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.802262    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"
Sep 16 19:08:48.810991 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.810970    2552 scope.go:117] "RemoveContainer" containerID="b478c6478f321c9fb85b999a48d0dcea5ec102aaca4dba6e51d6dce9192274dd"
Sep 16 19:08:48.819410 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.819390    2552 scope.go:117] "RemoveContainer" containerID="9322a514569e7c8657d2731cf7feddfbbda36296e663daa5e46244e36ec1c60a"
Sep 16 19:08:48.828793 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.828767    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"]
Sep 16 19:08:48.834553 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:48.834508    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp"]
Sep 16 19:08:50.625941 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.625895    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" path="/var/lib/kubelet/pods/e00f1837-9a15-4eff-a14e-cade78dbe4ab/volumes"
Sep 16 19:08:50.658130 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.658075    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"]
Sep 16 19:08:50.658520 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.658500    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerName="kserve-container"
Sep 16 19:08:50.658604 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.658522    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerName="kserve-container"
Sep 16 19:08:50.658604 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.658555    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerName="storage-initializer"
Sep 16 19:08:50.658604 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.658564    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerName="storage-initializer"
Sep 16 19:08:50.658604 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.658578    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerName="kube-rbac-proxy"
Sep 16 19:08:50.658604 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.658588    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerName="kube-rbac-proxy"
Sep 16 19:08:50.658865 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.658672    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerName="kube-rbac-proxy"
Sep 16 19:08:50.658865 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.658691    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e00f1837-9a15-4eff-a14e-cade78dbe4ab" containerName="kserve-container"
Sep 16 19:08:50.662604 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.662577    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:50.664731 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.664703    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 16 19:08:50.664874 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.664703    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 16 19:08:50.679420 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.679386    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"]
Sep 16 19:08:50.761900 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.761854    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:50.762120 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.761966    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:50.762120 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.762005    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9bzn\" (UniqueName: \"kubernetes.io/projected/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kube-api-access-n9bzn\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:50.762120 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.762029    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:50.762120 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.762055    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:50.862970 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.862928    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9bzn\" (UniqueName: \"kubernetes.io/projected/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kube-api-access-n9bzn\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:50.862970 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.862973    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:50.863280 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.862997    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:50.863280 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.863019    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:50.863280 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.863076    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:50.863280 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:08:50.863244    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 16 19:08:50.863506 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:08:50.863344    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-proxy-tls podName:55c0ff8a-8c5f-4af3-9d93-9079acdf588d nodeName:}" failed. No retries permitted until 2026-09-16 19:08:51.363306071 +0000 UTC m=+1727.257493340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8" (UID: "55c0ff8a-8c5f-4af3-9d93-9079acdf588d") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 16 19:08:50.863506 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.863436    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:50.863798 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.863776    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:50.863870 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.863782    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:50.872902 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:50.872868    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9bzn\" (UniqueName: \"kubernetes.io/projected/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kube-api-access-n9bzn\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:51.368140 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:51.368075    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:51.370557 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:51.370532    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:51.572975 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:51.572931    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:08:51.849225 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:51.849191    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"]
Sep 16 19:08:51.851991 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:08:51.851962    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c0ff8a_8c5f_4af3_9d93_9079acdf588d.slice/crio-d447a5c6643f115fa2ce4497c374d3ca07d9f167e0f2efedfb66ec169b98ab60 WatchSource:0}: Error finding container d447a5c6643f115fa2ce4497c374d3ca07d9f167e0f2efedfb66ec169b98ab60: Status 404 returned error can't find the container with id d447a5c6643f115fa2ce4497c374d3ca07d9f167e0f2efedfb66ec169b98ab60
Sep 16 19:08:52.733260 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:52.733201    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h" podUID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:08:52.823487 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:52.823450    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8" event={"ID":"55c0ff8a-8c5f-4af3-9d93-9079acdf588d","Type":"ContainerStarted","Data":"ef2a220cb8555c16420e329d7224a994bfe7f6af247d5b015350ad3aa5908072"}
Sep 16 19:08:52.823658 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:08:52.823510    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8" event={"ID":"55c0ff8a-8c5f-4af3-9d93-9079acdf588d","Type":"ContainerStarted","Data":"d447a5c6643f115fa2ce4497c374d3ca07d9f167e0f2efedfb66ec169b98ab60"}
Sep 16 19:09:01.620584 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:01.620482    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:09:01.621009 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:09:01.620657    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:09:01.875230 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:01.875192    2552 generic.go:356] "Generic (PLEG): container finished" podID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerID="ef2a220cb8555c16420e329d7224a994bfe7f6af247d5b015350ad3aa5908072" exitCode=0
Sep 16 19:09:01.875230 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:01.875236    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8" event={"ID":"55c0ff8a-8c5f-4af3-9d93-9079acdf588d","Type":"ContainerDied","Data":"ef2a220cb8555c16420e329d7224a994bfe7f6af247d5b015350ad3aa5908072"}
Sep 16 19:09:02.648430 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:02.648396    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:09:02.882730 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:02.882691    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8" event={"ID":"55c0ff8a-8c5f-4af3-9d93-9079acdf588d","Type":"ContainerStarted","Data":"6f375956f8102c1213806fe6aea1ce7ac63f3c7a7c9b7d3fe019306421ea5573"}
Sep 16 19:09:02.882934 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:02.882764    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8" event={"ID":"55c0ff8a-8c5f-4af3-9d93-9079acdf588d","Type":"ContainerStarted","Data":"cb9b6b51c63407325ef1237bf1e2bbff88b2a378bb0d6eb0ad31d545153f6cbe"}
Sep 16 19:09:02.883010 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:02.882997    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:09:02.903308 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:02.903181    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8" podStartSLOduration=3.574944378 podStartE2EDuration="12.903155103s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.328210725s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:02.901588312 +0000 UTC m=+1738.795775580" watchObservedRunningTime="2026-09-16 19:09:02.903155103 +0000 UTC m=+1738.797342376"
Sep 16 19:09:03.887314 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:03.887268    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:09:03.889165 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:03.889121    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8" podUID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 16 19:09:04.891992 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:04.891949    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8" podUID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 16 19:09:08.700746 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:08.700705    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"]
Sep 16 19:09:08.701294 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:08.700966    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h" podUID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerName="kserve-container" containerID="cri-o://e4a9a858abd53eb4a526ebfb4801721ed0581679692bd1b9588f30cfa3fbb4e6" gracePeriod=30
Sep 16 19:09:08.701294 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:08.701038    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h" podUID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerName="kube-rbac-proxy" containerID="cri-o://a151d46a1a329a42aafb7628166aeea3fb24ec4ef6ee18daac65c89fb5bd6059" gracePeriod=30
Sep 16 19:09:08.914914 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:08.914871    2552 generic.go:356] "Generic (PLEG): container finished" podID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerID="a151d46a1a329a42aafb7628166aeea3fb24ec4ef6ee18daac65c89fb5bd6059" exitCode=2
Sep 16 19:09:08.915103 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:08.914941    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h" event={"ID":"106aa30b-43e8-4356-94fa-0bfa18dfaad5","Type":"ContainerDied","Data":"a151d46a1a329a42aafb7628166aeea3fb24ec4ef6ee18daac65c89fb5bd6059"}
Sep 16 19:09:09.897923 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:09.897890    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:09:09.898473 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:09.898445    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8" podUID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 16 19:09:12.639037 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:12.638992    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h" podUID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 16 19:09:12.646315 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:12.646274    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h" podUID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.40:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:09:15.620638 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:15.620601    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:09:15.621207 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:09:15.620802    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:09:16.954632 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:16.954602    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:09:16.958670 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:16.958634    2552 generic.go:356] "Generic (PLEG): container finished" podID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerID="e4a9a858abd53eb4a526ebfb4801721ed0581679692bd1b9588f30cfa3fbb4e6" exitCode=0
Sep 16 19:09:16.958845 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:16.958689    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"
Sep 16 19:09:16.958845 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:16.958711    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h" event={"ID":"106aa30b-43e8-4356-94fa-0bfa18dfaad5","Type":"ContainerDied","Data":"e4a9a858abd53eb4a526ebfb4801721ed0581679692bd1b9588f30cfa3fbb4e6"}
Sep 16 19:09:16.958845 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:16.958783    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h" event={"ID":"106aa30b-43e8-4356-94fa-0bfa18dfaad5","Type":"ContainerDied","Data":"260a900fbe55791df6239c988bf4beed5babaa1a250a53b73e078d06ffb1b900"}
Sep 16 19:09:16.958845 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:16.958822    2552 scope.go:117] "RemoveContainer" containerID="a151d46a1a329a42aafb7628166aeea3fb24ec4ef6ee18daac65c89fb5bd6059"
Sep 16 19:09:16.967417 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:16.967386    2552 scope.go:117] "RemoveContainer" containerID="e4a9a858abd53eb4a526ebfb4801721ed0581679692bd1b9588f30cfa3fbb4e6"
Sep 16 19:09:16.976410 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:16.976249    2552 scope.go:117] "RemoveContainer" containerID="e94d0db3c3dfb13ff46a1c9c7211952c12534168da04d27af26a253ab42175aa"
Sep 16 19:09:16.986348 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:16.986316    2552 scope.go:117] "RemoveContainer" containerID="a151d46a1a329a42aafb7628166aeea3fb24ec4ef6ee18daac65c89fb5bd6059"
Sep 16 19:09:16.986739 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:09:16.986715    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a151d46a1a329a42aafb7628166aeea3fb24ec4ef6ee18daac65c89fb5bd6059\": container with ID starting with a151d46a1a329a42aafb7628166aeea3fb24ec4ef6ee18daac65c89fb5bd6059 not found: ID does not exist" containerID="a151d46a1a329a42aafb7628166aeea3fb24ec4ef6ee18daac65c89fb5bd6059"
Sep 16 19:09:16.986831 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:16.986749    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a151d46a1a329a42aafb7628166aeea3fb24ec4ef6ee18daac65c89fb5bd6059"} err="failed to get container status \"a151d46a1a329a42aafb7628166aeea3fb24ec4ef6ee18daac65c89fb5bd6059\": rpc error: code = NotFound desc = could not find container \"a151d46a1a329a42aafb7628166aeea3fb24ec4ef6ee18daac65c89fb5bd6059\": container with ID starting with a151d46a1a329a42aafb7628166aeea3fb24ec4ef6ee18daac65c89fb5bd6059 not found: ID does not exist"
Sep 16 19:09:16.986831 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:16.986768    2552 scope.go:117] "RemoveContainer" containerID="e4a9a858abd53eb4a526ebfb4801721ed0581679692bd1b9588f30cfa3fbb4e6"
Sep 16 19:09:16.987069 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:09:16.987053    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a9a858abd53eb4a526ebfb4801721ed0581679692bd1b9588f30cfa3fbb4e6\": container with ID starting with e4a9a858abd53eb4a526ebfb4801721ed0581679692bd1b9588f30cfa3fbb4e6 not found: ID does not exist" containerID="e4a9a858abd53eb4a526ebfb4801721ed0581679692bd1b9588f30cfa3fbb4e6"
Sep 16 19:09:16.987134 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:16.987077    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a9a858abd53eb4a526ebfb4801721ed0581679692bd1b9588f30cfa3fbb4e6"} err="failed to get container status \"e4a9a858abd53eb4a526ebfb4801721ed0581679692bd1b9588f30cfa3fbb4e6\": rpc error: code = NotFound desc = could not find container \"e4a9a858abd53eb4a526ebfb4801721ed0581679692bd1b9588f30cfa3fbb4e6\": container with ID starting with e4a9a858abd53eb4a526ebfb4801721ed0581679692bd1b9588f30cfa3fbb4e6 not found: ID does not exist"
Sep 16 19:09:16.987134 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:16.987111    2552 scope.go:117] "RemoveContainer" containerID="e94d0db3c3dfb13ff46a1c9c7211952c12534168da04d27af26a253ab42175aa"
Sep 16 19:09:16.987402 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:09:16.987376    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94d0db3c3dfb13ff46a1c9c7211952c12534168da04d27af26a253ab42175aa\": container with ID starting with e94d0db3c3dfb13ff46a1c9c7211952c12534168da04d27af26a253ab42175aa not found: ID does not exist" containerID="e94d0db3c3dfb13ff46a1c9c7211952c12534168da04d27af26a253ab42175aa"
Sep 16 19:09:16.987463 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:16.987406    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94d0db3c3dfb13ff46a1c9c7211952c12534168da04d27af26a253ab42175aa"} err="failed to get container status \"e94d0db3c3dfb13ff46a1c9c7211952c12534168da04d27af26a253ab42175aa\": rpc error: code = NotFound desc = could not find container \"e94d0db3c3dfb13ff46a1c9c7211952c12534168da04d27af26a253ab42175aa\": container with ID starting with e94d0db3c3dfb13ff46a1c9c7211952c12534168da04d27af26a253ab42175aa not found: ID does not exist"
Sep 16 19:09:17.102990 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.102890    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/106aa30b-43e8-4356-94fa-0bfa18dfaad5-cabundle-cert\") pod \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") "
Sep 16 19:09:17.102990 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.102959    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/106aa30b-43e8-4356-94fa-0bfa18dfaad5-proxy-tls\") pod \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") "
Sep 16 19:09:17.103209 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.102998    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kube-rbac-proxy-sar-config\") pod \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") "
Sep 16 19:09:17.103209 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.103068    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kserve-provision-location\") pod \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") "
Sep 16 19:09:17.103209 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.103120    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhtn8\" (UniqueName: \"kubernetes.io/projected/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kube-api-access-dhtn8\") pod \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\" (UID: \"106aa30b-43e8-4356-94fa-0bfa18dfaad5\") "
Sep 16 19:09:17.103361 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.103334    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106aa30b-43e8-4356-94fa-0bfa18dfaad5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "106aa30b-43e8-4356-94fa-0bfa18dfaad5" (UID: "106aa30b-43e8-4356-94fa-0bfa18dfaad5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:17.103401 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.103379    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "106aa30b-43e8-4356-94fa-0bfa18dfaad5" (UID: "106aa30b-43e8-4356-94fa-0bfa18dfaad5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:17.103445 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.103416    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "106aa30b-43e8-4356-94fa-0bfa18dfaad5" (UID: "106aa30b-43e8-4356-94fa-0bfa18dfaad5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:17.105287 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.105249    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/106aa30b-43e8-4356-94fa-0bfa18dfaad5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "106aa30b-43e8-4356-94fa-0bfa18dfaad5" (UID: "106aa30b-43e8-4356-94fa-0bfa18dfaad5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:17.105423 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.105408    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kube-api-access-dhtn8" (OuterVolumeSpecName: "kube-api-access-dhtn8") pod "106aa30b-43e8-4356-94fa-0bfa18dfaad5" (UID: "106aa30b-43e8-4356-94fa-0bfa18dfaad5"). InnerVolumeSpecName "kube-api-access-dhtn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:17.204484 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.204444    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:09:17.204484 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.204479    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhtn8\" (UniqueName: \"kubernetes.io/projected/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kube-api-access-dhtn8\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:09:17.204484 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.204494    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/106aa30b-43e8-4356-94fa-0bfa18dfaad5-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:09:17.204742 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.204507    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/106aa30b-43e8-4356-94fa-0bfa18dfaad5-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:09:17.204742 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.204518    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/106aa30b-43e8-4356-94fa-0bfa18dfaad5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:09:17.287923 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.287878    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"]
Sep 16 19:09:17.294124 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:17.294071    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h"]
Sep 16 19:09:18.629917 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:18.629874    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" path="/var/lib/kubelet/pods/106aa30b-43e8-4356-94fa-0bfa18dfaad5/volumes"
Sep 16 19:09:19.899273 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:19.899230    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:09:28.620534 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:28.620493    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:09:28.620944 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:09:28.620757    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:09:30.253937 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:30.253886    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"]
Sep 16 19:09:30.254374 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:30.254256    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8" podUID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerName="kube-rbac-proxy" containerID="cri-o://6f375956f8102c1213806fe6aea1ce7ac63f3c7a7c9b7d3fe019306421ea5573" gracePeriod=30
Sep 16 19:09:30.254374 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:30.254237    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8" podUID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerName="kserve-container" containerID="cri-o://cb9b6b51c63407325ef1237bf1e2bbff88b2a378bb0d6eb0ad31d545153f6cbe" gracePeriod=30
Sep 16 19:09:31.033848 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:31.033811    2552 generic.go:356] "Generic (PLEG): container finished" podID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerID="6f375956f8102c1213806fe6aea1ce7ac63f3c7a7c9b7d3fe019306421ea5573" exitCode=2
Sep 16 19:09:31.034036 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:31.033893    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8" event={"ID":"55c0ff8a-8c5f-4af3-9d93-9079acdf588d","Type":"ContainerDied","Data":"6f375956f8102c1213806fe6aea1ce7ac63f3c7a7c9b7d3fe019306421ea5573"}
Sep 16 19:09:31.912973 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:31.912941    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:09:32.018326 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.018218    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-proxy-tls\") pod \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") "
Sep 16 19:09:32.018326 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.018286    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kserve-provision-location\") pod \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") "
Sep 16 19:09:32.018326 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.018306    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-cabundle-cert\") pod \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") "
Sep 16 19:09:32.018696 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.018349    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9bzn\" (UniqueName: \"kubernetes.io/projected/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kube-api-access-n9bzn\") pod \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") "
Sep 16 19:09:32.018696 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.018370    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kube-rbac-proxy-sar-config\") pod \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\" (UID: \"55c0ff8a-8c5f-4af3-9d93-9079acdf588d\") "
Sep 16 19:09:32.018808 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.018736    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "55c0ff8a-8c5f-4af3-9d93-9079acdf588d" (UID: "55c0ff8a-8c5f-4af3-9d93-9079acdf588d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:32.018808 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.018788    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "55c0ff8a-8c5f-4af3-9d93-9079acdf588d" (UID: "55c0ff8a-8c5f-4af3-9d93-9079acdf588d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:32.019286 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.019262    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55c0ff8a-8c5f-4af3-9d93-9079acdf588d" (UID: "55c0ff8a-8c5f-4af3-9d93-9079acdf588d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:32.020486 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.020459    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kube-api-access-n9bzn" (OuterVolumeSpecName: "kube-api-access-n9bzn") pod "55c0ff8a-8c5f-4af3-9d93-9079acdf588d" (UID: "55c0ff8a-8c5f-4af3-9d93-9079acdf588d"). InnerVolumeSpecName "kube-api-access-n9bzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:32.020605 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.020562    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "55c0ff8a-8c5f-4af3-9d93-9079acdf588d" (UID: "55c0ff8a-8c5f-4af3-9d93-9079acdf588d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:32.040796 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.040748    2552 generic.go:356] "Generic (PLEG): container finished" podID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerID="cb9b6b51c63407325ef1237bf1e2bbff88b2a378bb0d6eb0ad31d545153f6cbe" exitCode=0
Sep 16 19:09:32.040936 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.040813    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"
Sep 16 19:09:32.040936 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.040809    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8" event={"ID":"55c0ff8a-8c5f-4af3-9d93-9079acdf588d","Type":"ContainerDied","Data":"cb9b6b51c63407325ef1237bf1e2bbff88b2a378bb0d6eb0ad31d545153f6cbe"}
Sep 16 19:09:32.041032 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.040991    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8" event={"ID":"55c0ff8a-8c5f-4af3-9d93-9079acdf588d","Type":"ContainerDied","Data":"d447a5c6643f115fa2ce4497c374d3ca07d9f167e0f2efedfb66ec169b98ab60"}
Sep 16 19:09:32.041032 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.041029    2552 scope.go:117] "RemoveContainer" containerID="6f375956f8102c1213806fe6aea1ce7ac63f3c7a7c9b7d3fe019306421ea5573"
Sep 16 19:09:32.050214 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.050194    2552 scope.go:117] "RemoveContainer" containerID="cb9b6b51c63407325ef1237bf1e2bbff88b2a378bb0d6eb0ad31d545153f6cbe"
Sep 16 19:09:32.058623 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.058601    2552 scope.go:117] "RemoveContainer" containerID="ef2a220cb8555c16420e329d7224a994bfe7f6af247d5b015350ad3aa5908072"
Sep 16 19:09:32.066153 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.066117    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"]
Sep 16 19:09:32.067368 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.067350    2552 scope.go:117] "RemoveContainer" containerID="6f375956f8102c1213806fe6aea1ce7ac63f3c7a7c9b7d3fe019306421ea5573"
Sep 16 19:09:32.067720 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:09:32.067700    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f375956f8102c1213806fe6aea1ce7ac63f3c7a7c9b7d3fe019306421ea5573\": container with ID starting with 6f375956f8102c1213806fe6aea1ce7ac63f3c7a7c9b7d3fe019306421ea5573 not found: ID does not exist" containerID="6f375956f8102c1213806fe6aea1ce7ac63f3c7a7c9b7d3fe019306421ea5573"
Sep 16 19:09:32.067797 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.067728    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f375956f8102c1213806fe6aea1ce7ac63f3c7a7c9b7d3fe019306421ea5573"} err="failed to get container status \"6f375956f8102c1213806fe6aea1ce7ac63f3c7a7c9b7d3fe019306421ea5573\": rpc error: code = NotFound desc = could not find container \"6f375956f8102c1213806fe6aea1ce7ac63f3c7a7c9b7d3fe019306421ea5573\": container with ID starting with 6f375956f8102c1213806fe6aea1ce7ac63f3c7a7c9b7d3fe019306421ea5573 not found: ID does not exist"
Sep 16 19:09:32.067797 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.067747    2552 scope.go:117] "RemoveContainer" containerID="cb9b6b51c63407325ef1237bf1e2bbff88b2a378bb0d6eb0ad31d545153f6cbe"
Sep 16 19:09:32.068037 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:09:32.068009    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9b6b51c63407325ef1237bf1e2bbff88b2a378bb0d6eb0ad31d545153f6cbe\": container with ID starting with cb9b6b51c63407325ef1237bf1e2bbff88b2a378bb0d6eb0ad31d545153f6cbe not found: ID does not exist" containerID="cb9b6b51c63407325ef1237bf1e2bbff88b2a378bb0d6eb0ad31d545153f6cbe"
Sep 16 19:09:32.068079 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.068045    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9b6b51c63407325ef1237bf1e2bbff88b2a378bb0d6eb0ad31d545153f6cbe"} err="failed to get container status \"cb9b6b51c63407325ef1237bf1e2bbff88b2a378bb0d6eb0ad31d545153f6cbe\": rpc error: code = NotFound desc = could not find container \"cb9b6b51c63407325ef1237bf1e2bbff88b2a378bb0d6eb0ad31d545153f6cbe\": container with ID starting with cb9b6b51c63407325ef1237bf1e2bbff88b2a378bb0d6eb0ad31d545153f6cbe not found: ID does not exist"
Sep 16 19:09:32.068079 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.068061    2552 scope.go:117] "RemoveContainer" containerID="ef2a220cb8555c16420e329d7224a994bfe7f6af247d5b015350ad3aa5908072"
Sep 16 19:09:32.068367 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:09:32.068348    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef2a220cb8555c16420e329d7224a994bfe7f6af247d5b015350ad3aa5908072\": container with ID starting with ef2a220cb8555c16420e329d7224a994bfe7f6af247d5b015350ad3aa5908072 not found: ID does not exist" containerID="ef2a220cb8555c16420e329d7224a994bfe7f6af247d5b015350ad3aa5908072"
Sep 16 19:09:32.068413 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.068375    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2a220cb8555c16420e329d7224a994bfe7f6af247d5b015350ad3aa5908072"} err="failed to get container status \"ef2a220cb8555c16420e329d7224a994bfe7f6af247d5b015350ad3aa5908072\": rpc error: code = NotFound desc = could not find container \"ef2a220cb8555c16420e329d7224a994bfe7f6af247d5b015350ad3aa5908072\": container with ID starting with ef2a220cb8555c16420e329d7224a994bfe7f6af247d5b015350ad3aa5908072 not found: ID does not exist"
Sep 16 19:09:32.071048 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.071016    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8"]
Sep 16 19:09:32.119458 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.119418    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:09:32.119458 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.119454    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:09:32.119458 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.119465    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:09:32.119687 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.119476    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9bzn\" (UniqueName: \"kubernetes.io/projected/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kube-api-access-n9bzn\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:09:32.119687 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.119485    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55c0ff8a-8c5f-4af3-9d93-9079acdf588d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:09:32.625566 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:32.625527    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" path="/var/lib/kubelet/pods/55c0ff8a-8c5f-4af3-9d93-9079acdf588d/volumes"
Sep 16 19:09:35.550522 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550483    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"]
Sep 16 19:09:35.550918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550800    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerName="storage-initializer"
Sep 16 19:09:35.550918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550811    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerName="storage-initializer"
Sep 16 19:09:35.550918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550821    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerName="kserve-container"
Sep 16 19:09:35.550918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550828    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerName="kserve-container"
Sep 16 19:09:35.550918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550836    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerName="kube-rbac-proxy"
Sep 16 19:09:35.550918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550842    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerName="kube-rbac-proxy"
Sep 16 19:09:35.550918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550854    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerName="kube-rbac-proxy"
Sep 16 19:09:35.550918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550859    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerName="kube-rbac-proxy"
Sep 16 19:09:35.550918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550865    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerName="storage-initializer"
Sep 16 19:09:35.550918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550870    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerName="storage-initializer"
Sep 16 19:09:35.550918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550878    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerName="kserve-container"
Sep 16 19:09:35.550918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550883    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerName="kserve-container"
Sep 16 19:09:35.551313 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550937    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerName="kube-rbac-proxy"
Sep 16 19:09:35.551313 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550945    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerName="kserve-container"
Sep 16 19:09:35.551313 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550951    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c0ff8a-8c5f-4af3-9d93-9079acdf588d" containerName="kube-rbac-proxy"
Sep 16 19:09:35.551313 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.550958    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="106aa30b-43e8-4356-94fa-0bfa18dfaad5" containerName="kserve-container"
Sep 16 19:09:35.554225 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.554201    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.556520 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.556488    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:09:35.556520 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.556490    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:09:35.556728 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.556538    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:09:35.556728 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.556586    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:09:35.556806 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.556756    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:09:35.556977 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.556958    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:09:35.557033 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.556958    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:09:35.565164 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.565129    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"]
Sep 16 19:09:35.645675 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.645624    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f066541-420e-4f30-8be3-4cff06947d49-cabundle-cert\") pod \"isvc-canary-create-predictor-68cd88b8-vwczt\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.645862 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.645691    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntlzq\" (UniqueName: \"kubernetes.io/projected/7f066541-420e-4f30-8be3-4cff06947d49-kube-api-access-ntlzq\") pod \"isvc-canary-create-predictor-68cd88b8-vwczt\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.645862 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.645784    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f066541-420e-4f30-8be3-4cff06947d49-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68cd88b8-vwczt\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.645862 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.645830    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f066541-420e-4f30-8be3-4cff06947d49-kserve-provision-location\") pod \"isvc-canary-create-predictor-68cd88b8-vwczt\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.645967 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.645864    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f066541-420e-4f30-8be3-4cff06947d49-proxy-tls\") pod \"isvc-canary-create-predictor-68cd88b8-vwczt\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.747314 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.747266    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f066541-420e-4f30-8be3-4cff06947d49-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68cd88b8-vwczt\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.747523 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.747322    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f066541-420e-4f30-8be3-4cff06947d49-kserve-provision-location\") pod \"isvc-canary-create-predictor-68cd88b8-vwczt\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.747523 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.747350    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f066541-420e-4f30-8be3-4cff06947d49-proxy-tls\") pod \"isvc-canary-create-predictor-68cd88b8-vwczt\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.747523 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.747382    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f066541-420e-4f30-8be3-4cff06947d49-cabundle-cert\") pod \"isvc-canary-create-predictor-68cd88b8-vwczt\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.747721 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.747596    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntlzq\" (UniqueName: \"kubernetes.io/projected/7f066541-420e-4f30-8be3-4cff06947d49-kube-api-access-ntlzq\") pod \"isvc-canary-create-predictor-68cd88b8-vwczt\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.747790 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.747737    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f066541-420e-4f30-8be3-4cff06947d49-kserve-provision-location\") pod \"isvc-canary-create-predictor-68cd88b8-vwczt\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.748117 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.748069    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f066541-420e-4f30-8be3-4cff06947d49-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68cd88b8-vwczt\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.748205 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.748111    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f066541-420e-4f30-8be3-4cff06947d49-cabundle-cert\") pod \"isvc-canary-create-predictor-68cd88b8-vwczt\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.749995 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.749972    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f066541-420e-4f30-8be3-4cff06947d49-proxy-tls\") pod \"isvc-canary-create-predictor-68cd88b8-vwczt\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.757286 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.757250    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntlzq\" (UniqueName: \"kubernetes.io/projected/7f066541-420e-4f30-8be3-4cff06947d49-kube-api-access-ntlzq\") pod \"isvc-canary-create-predictor-68cd88b8-vwczt\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:35.866055 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:35.865952    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:36.141766 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:36.141728    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"]
Sep 16 19:09:36.145835 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:09:36.145803    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f066541_420e_4f30_8be3_4cff06947d49.slice/crio-50f56a9f4a589a0d47cbaec5c8e176aa756c3680db66cc89ed9426f5bf803879 WatchSource:0}: Error finding container 50f56a9f4a589a0d47cbaec5c8e176aa756c3680db66cc89ed9426f5bf803879: Status 404 returned error can't find the container with id 50f56a9f4a589a0d47cbaec5c8e176aa756c3680db66cc89ed9426f5bf803879
Sep 16 19:09:37.066075 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:37.066035    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" event={"ID":"7f066541-420e-4f30-8be3-4cff06947d49","Type":"ContainerStarted","Data":"c657bd8aaa8dc9fb63dbaa3115afe041edc5ccd42e03cb21239dde27324eb0fe"}
Sep 16 19:09:37.066523 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:37.066130    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" event={"ID":"7f066541-420e-4f30-8be3-4cff06947d49","Type":"ContainerStarted","Data":"50f56a9f4a589a0d47cbaec5c8e176aa756c3680db66cc89ed9426f5bf803879"}
Sep 16 19:09:40.087836 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:40.087796    2552 generic.go:356] "Generic (PLEG): container finished" podID="7f066541-420e-4f30-8be3-4cff06947d49" containerID="c657bd8aaa8dc9fb63dbaa3115afe041edc5ccd42e03cb21239dde27324eb0fe" exitCode=0
Sep 16 19:09:40.088252 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:40.087864    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" event={"ID":"7f066541-420e-4f30-8be3-4cff06947d49","Type":"ContainerDied","Data":"c657bd8aaa8dc9fb63dbaa3115afe041edc5ccd42e03cb21239dde27324eb0fe"}
Sep 16 19:09:40.089062 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:40.089043    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:09:43.620587 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:43.620143    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:09:43.620587 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:09:43.620384    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:09:54.183525 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:54.183474    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" event={"ID":"7f066541-420e-4f30-8be3-4cff06947d49","Type":"ContainerStarted","Data":"82e3e7de0ff89180631e77063250ed0a0f98ba96c6a4097f95b53cbae3d29182"}
Sep 16 19:09:54.184068 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:54.183596    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" event={"ID":"7f066541-420e-4f30-8be3-4cff06947d49","Type":"ContainerStarted","Data":"9082392b6eeaa0507134595077cd8342ad6a2296fc79bba77191c0f275ea5357"}
Sep 16 19:09:54.184068 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:54.183835    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:54.205299 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:54.205236    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" podStartSLOduration=2.068864667 podStartE2EDuration="19.205216231s" totalImagesPullingTime="13.373534418s" totalInitContainerRuntime="3.762817146s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:54.204051905 +0000 UTC m=+1790.098239178" watchObservedRunningTime="2026-09-16 19:09:54.205216231 +0000 UTC m=+1790.099403499"
Sep 16 19:09:55.188364 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:55.188322    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:09:55.189718 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:55.189686    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 19:09:56.193253 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:56.193214    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 19:09:57.620363 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:09:57.620330    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:09:57.620756 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:09:57.620546    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:10:01.197646 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:01.197612    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:10:01.198336 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:01.198307    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 19:10:09.620491 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:09.620451    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:10:09.620944 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:10:09.620649    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:10:11.198395 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:11.198355    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 19:10:21.198398 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:21.198342    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 19:10:23.619595 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:23.619560    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:10:23.619993 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:10:23.619755    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:10:31.198606 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:31.198563    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 19:10:38.626379 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:38.626347    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:10:38.626861 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:10:38.626550    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:10:41.198780 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:41.198739    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 19:10:49.255937 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.255890    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"]
Sep 16 19:10:49.259614 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.259590    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:49.262009 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.261982    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 16 19:10:49.262009 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.261982    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 16 19:10:49.272562 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.272527    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"]
Sep 16 19:10:49.395971 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.395927    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:49.396179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.395980    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:49.396179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.396009    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cjrb\" (UniqueName: \"kubernetes.io/projected/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kube-api-access-7cjrb\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:49.396179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.396036    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b04d1fd-faf8-4b33-9c09-1d1da0244743-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:49.396179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.396064    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b04d1fd-faf8-4b33-9c09-1d1da0244743-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:49.496940 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.496890    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:49.496940 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.496949    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:49.497278 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.496973    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cjrb\" (UniqueName: \"kubernetes.io/projected/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kube-api-access-7cjrb\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:49.497278 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.496999    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b04d1fd-faf8-4b33-9c09-1d1da0244743-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:49.497278 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.497027    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b04d1fd-faf8-4b33-9c09-1d1da0244743-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:49.497278 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:10:49.497267    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 16 19:10:49.497486 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:10:49.497369    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b04d1fd-faf8-4b33-9c09-1d1da0244743-proxy-tls podName:9b04d1fd-faf8-4b33-9c09-1d1da0244743 nodeName:}" failed. No retries permitted until 2026-09-16 19:10:49.997343596 +0000 UTC m=+1845.891530844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9b04d1fd-faf8-4b33-9c09-1d1da0244743-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" (UID: "9b04d1fd-faf8-4b33-9c09-1d1da0244743") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 16 19:10:49.497486 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.497392    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:49.497805 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.497782    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:49.497882 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.497865    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b04d1fd-faf8-4b33-9c09-1d1da0244743-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:49.512487 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:49.512412    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cjrb\" (UniqueName: \"kubernetes.io/projected/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kube-api-access-7cjrb\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:50.000816 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:50.000774    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b04d1fd-faf8-4b33-9c09-1d1da0244743-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:50.001002 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:10:50.000934    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 16 19:10:50.001002 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:10:50.000999    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b04d1fd-faf8-4b33-9c09-1d1da0244743-proxy-tls podName:9b04d1fd-faf8-4b33-9c09-1d1da0244743 nodeName:}" failed. No retries permitted until 2026-09-16 19:10:51.000982941 +0000 UTC m=+1846.895170208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9b04d1fd-faf8-4b33-9c09-1d1da0244743-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" (UID: "9b04d1fd-faf8-4b33-9c09-1d1da0244743") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 16 19:10:51.010795 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:51.010739    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b04d1fd-faf8-4b33-9c09-1d1da0244743-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:51.013487 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:51.013447    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b04d1fd-faf8-4b33-9c09-1d1da0244743-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:51.071106 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:51.071055    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:51.198684 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:51.198643    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 19:10:51.347787 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:51.347691    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"]
Sep 16 19:10:51.350814 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:10:51.350785    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b04d1fd_faf8_4b33_9c09_1d1da0244743.slice/crio-0ea17a94c1b8c4d67b4efbc8f2a1be753a59ecff02047b95f304103f9c9003f9 WatchSource:0}: Error finding container 0ea17a94c1b8c4d67b4efbc8f2a1be753a59ecff02047b95f304103f9c9003f9: Status 404 returned error can't find the container with id 0ea17a94c1b8c4d67b4efbc8f2a1be753a59ecff02047b95f304103f9c9003f9
Sep 16 19:10:51.450681 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:51.450637    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" event={"ID":"9b04d1fd-faf8-4b33-9c09-1d1da0244743","Type":"ContainerStarted","Data":"fb1ea8cc6b4afc09245092fd9151a02add0e330084ad98458d01ec76b5d0bbb1"}
Sep 16 19:10:51.450833 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:51.450730    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" event={"ID":"9b04d1fd-faf8-4b33-9c09-1d1da0244743","Type":"ContainerStarted","Data":"0ea17a94c1b8c4d67b4efbc8f2a1be753a59ecff02047b95f304103f9c9003f9"}
Sep 16 19:10:52.619907 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:52.619876    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:10:52.622231 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:10:52.620067    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:10:55.472410 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:55.472374    2552 generic.go:356] "Generic (PLEG): container finished" podID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerID="fb1ea8cc6b4afc09245092fd9151a02add0e330084ad98458d01ec76b5d0bbb1" exitCode=0
Sep 16 19:10:55.472813 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:55.472457    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" event={"ID":"9b04d1fd-faf8-4b33-9c09-1d1da0244743","Type":"ContainerDied","Data":"fb1ea8cc6b4afc09245092fd9151a02add0e330084ad98458d01ec76b5d0bbb1"}
Sep 16 19:10:56.480023 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:56.479987    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" event={"ID":"9b04d1fd-faf8-4b33-9c09-1d1da0244743","Type":"ContainerStarted","Data":"4ec35421e14086b9a97eb45855a7fe39a7fc3261e991a3d4265aac4473e3d217"}
Sep 16 19:10:56.480456 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:56.480054    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" event={"ID":"9b04d1fd-faf8-4b33-9c09-1d1da0244743","Type":"ContainerStarted","Data":"ed93c5fa48c89f0045d2a40e97a6ce350cc564e63099172ef512542f4d694060"}
Sep 16 19:10:56.480456 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:56.480359    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:56.480571 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:56.480487    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:10:56.481824 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:56.481796    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:10:56.504296 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:56.504234    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" podStartSLOduration=3.6502764819999998 podStartE2EDuration="7.504217351s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.853940869s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:56.502854736 +0000 UTC m=+1852.397042006" watchObservedRunningTime="2026-09-16 19:10:56.504217351 +0000 UTC m=+1852.398404617"
Sep 16 19:10:57.484987 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:10:57.484948    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:11:01.198609 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:01.198565    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 19:11:02.489834 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:02.489798    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:11:02.490476 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:02.490448    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:11:03.620366 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:03.620330    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:11:03.620769 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:11:03.620516    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:11:11.199518 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:11.199485    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:11:12.491181 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:12.491130    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:11:15.555506 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:15.555457    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"]
Sep 16 19:11:15.555958 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:15.555785    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kserve-container" containerID="cri-o://9082392b6eeaa0507134595077cd8342ad6a2296fc79bba77191c0f275ea5357" gracePeriod=30
Sep 16 19:11:15.555958 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:15.555812    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kube-rbac-proxy" containerID="cri-o://82e3e7de0ff89180631e77063250ed0a0f98ba96c6a4097f95b53cbae3d29182" gracePeriod=30
Sep 16 19:11:16.193644 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:16.193598    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 16 19:11:16.574568 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:16.574470    2552 generic.go:356] "Generic (PLEG): container finished" podID="7f066541-420e-4f30-8be3-4cff06947d49" containerID="82e3e7de0ff89180631e77063250ed0a0f98ba96c6a4097f95b53cbae3d29182" exitCode=2
Sep 16 19:11:16.574568 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:16.574549    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" event={"ID":"7f066541-420e-4f30-8be3-4cff06947d49","Type":"ContainerDied","Data":"82e3e7de0ff89180631e77063250ed0a0f98ba96c6a4097f95b53cbae3d29182"}
Sep 16 19:11:17.620455 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:17.620417    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:11:17.620934 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:11:17.620659    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:11:20.889397 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:20.889356    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"]
Sep 16 19:11:20.893030 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:20.893006    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:20.895363 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:20.895336    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:11:20.895495 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:20.895337    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:11:20.902508 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:20.902473    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"]
Sep 16 19:11:20.957673 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:20.957629    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f69e899-2c60-4d92-b39a-5f410344fd0a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-674955ff8d-njhv9\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:20.957673 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:20.957674    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvwrs\" (UniqueName: \"kubernetes.io/projected/4f69e899-2c60-4d92-b39a-5f410344fd0a-kube-api-access-tvwrs\") pod \"isvc-canary-promote-predictor-674955ff8d-njhv9\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:20.957955 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:20.957773    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f69e899-2c60-4d92-b39a-5f410344fd0a-proxy-tls\") pod \"isvc-canary-promote-predictor-674955ff8d-njhv9\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:20.957955 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:20.957851    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f69e899-2c60-4d92-b39a-5f410344fd0a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-674955ff8d-njhv9\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:20.957955 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:20.957875    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f69e899-2c60-4d92-b39a-5f410344fd0a-cabundle-cert\") pod \"isvc-canary-promote-predictor-674955ff8d-njhv9\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:21.058846 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.058807    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f69e899-2c60-4d92-b39a-5f410344fd0a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-674955ff8d-njhv9\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:21.058846 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.058856    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f69e899-2c60-4d92-b39a-5f410344fd0a-cabundle-cert\") pod \"isvc-canary-promote-predictor-674955ff8d-njhv9\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:21.059110 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.058909    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f69e899-2c60-4d92-b39a-5f410344fd0a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-674955ff8d-njhv9\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:21.059110 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.058941    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvwrs\" (UniqueName: \"kubernetes.io/projected/4f69e899-2c60-4d92-b39a-5f410344fd0a-kube-api-access-tvwrs\") pod \"isvc-canary-promote-predictor-674955ff8d-njhv9\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:21.059110 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.059002    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f69e899-2c60-4d92-b39a-5f410344fd0a-proxy-tls\") pod \"isvc-canary-promote-predictor-674955ff8d-njhv9\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:21.059366 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.059294    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f69e899-2c60-4d92-b39a-5f410344fd0a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-674955ff8d-njhv9\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:21.059616 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.059594    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f69e899-2c60-4d92-b39a-5f410344fd0a-cabundle-cert\") pod \"isvc-canary-promote-predictor-674955ff8d-njhv9\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:21.059679 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.059599    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f69e899-2c60-4d92-b39a-5f410344fd0a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-674955ff8d-njhv9\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:21.061628 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.061599    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f69e899-2c60-4d92-b39a-5f410344fd0a-proxy-tls\") pod \"isvc-canary-promote-predictor-674955ff8d-njhv9\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:21.069145 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.069122    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvwrs\" (UniqueName: \"kubernetes.io/projected/4f69e899-2c60-4d92-b39a-5f410344fd0a-kube-api-access-tvwrs\") pod \"isvc-canary-promote-predictor-674955ff8d-njhv9\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:21.204031 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.204000    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:21.208546 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.208521    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:11:21.260638 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.260526    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f066541-420e-4f30-8be3-4cff06947d49-proxy-tls\") pod \"7f066541-420e-4f30-8be3-4cff06947d49\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") "
Sep 16 19:11:21.260638 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.260588    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f066541-420e-4f30-8be3-4cff06947d49-cabundle-cert\") pod \"7f066541-420e-4f30-8be3-4cff06947d49\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") "
Sep 16 19:11:21.260638 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.260612    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f066541-420e-4f30-8be3-4cff06947d49-kube-rbac-proxy-sar-config\") pod \"7f066541-420e-4f30-8be3-4cff06947d49\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") "
Sep 16 19:11:21.260903 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.260717    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntlzq\" (UniqueName: \"kubernetes.io/projected/7f066541-420e-4f30-8be3-4cff06947d49-kube-api-access-ntlzq\") pod \"7f066541-420e-4f30-8be3-4cff06947d49\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") "
Sep 16 19:11:21.261586 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.261130    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f066541-420e-4f30-8be3-4cff06947d49-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f066541-420e-4f30-8be3-4cff06947d49" (UID: "7f066541-420e-4f30-8be3-4cff06947d49"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:21.261586 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.261446    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f066541-420e-4f30-8be3-4cff06947d49-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f066541-420e-4f30-8be3-4cff06947d49" (UID: "7f066541-420e-4f30-8be3-4cff06947d49"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:21.264785 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.264719    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f066541-420e-4f30-8be3-4cff06947d49-kube-api-access-ntlzq" (OuterVolumeSpecName: "kube-api-access-ntlzq") pod "7f066541-420e-4f30-8be3-4cff06947d49" (UID: "7f066541-420e-4f30-8be3-4cff06947d49"). InnerVolumeSpecName "kube-api-access-ntlzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:21.266472 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.266422    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f066541-420e-4f30-8be3-4cff06947d49-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f066541-420e-4f30-8be3-4cff06947d49" (UID: "7f066541-420e-4f30-8be3-4cff06947d49"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:21.361295 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.361240    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f066541-420e-4f30-8be3-4cff06947d49-kserve-provision-location\") pod \"7f066541-420e-4f30-8be3-4cff06947d49\" (UID: \"7f066541-420e-4f30-8be3-4cff06947d49\") "
Sep 16 19:11:21.361493 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.361407    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntlzq\" (UniqueName: \"kubernetes.io/projected/7f066541-420e-4f30-8be3-4cff06947d49-kube-api-access-ntlzq\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:11:21.361493 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.361421    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f066541-420e-4f30-8be3-4cff06947d49-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:11:21.361493 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.361432    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f066541-420e-4f30-8be3-4cff06947d49-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:11:21.361493 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.361440    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f066541-420e-4f30-8be3-4cff06947d49-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:11:21.361629 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.361547    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f066541-420e-4f30-8be3-4cff06947d49-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f066541-420e-4f30-8be3-4cff06947d49" (UID: "7f066541-420e-4f30-8be3-4cff06947d49"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:21.461970 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.461885    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f066541-420e-4f30-8be3-4cff06947d49-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:11:21.485729 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.485698    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"]
Sep 16 19:11:21.488323 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:11:21.488294    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f69e899_2c60_4d92_b39a_5f410344fd0a.slice/crio-f346f225ae29b60f10213b2d3e0c263fc522aeb9318601cfe7480486903108d1 WatchSource:0}: Error finding container f346f225ae29b60f10213b2d3e0c263fc522aeb9318601cfe7480486903108d1: Status 404 returned error can't find the container with id f346f225ae29b60f10213b2d3e0c263fc522aeb9318601cfe7480486903108d1
Sep 16 19:11:21.608012 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.607966    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" event={"ID":"4f69e899-2c60-4d92-b39a-5f410344fd0a","Type":"ContainerStarted","Data":"0a8231552fbc75d5732b101e7d77e3bf8c6425a9696dc09123d0fa1f3077e05b"}
Sep 16 19:11:21.608210 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.608041    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" event={"ID":"4f69e899-2c60-4d92-b39a-5f410344fd0a","Type":"ContainerStarted","Data":"f346f225ae29b60f10213b2d3e0c263fc522aeb9318601cfe7480486903108d1"}
Sep 16 19:11:21.610157 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.610125    2552 generic.go:356] "Generic (PLEG): container finished" podID="7f066541-420e-4f30-8be3-4cff06947d49" containerID="9082392b6eeaa0507134595077cd8342ad6a2296fc79bba77191c0f275ea5357" exitCode=0
Sep 16 19:11:21.610288 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.610194    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" event={"ID":"7f066541-420e-4f30-8be3-4cff06947d49","Type":"ContainerDied","Data":"9082392b6eeaa0507134595077cd8342ad6a2296fc79bba77191c0f275ea5357"}
Sep 16 19:11:21.610288 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.610254    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" event={"ID":"7f066541-420e-4f30-8be3-4cff06947d49","Type":"ContainerDied","Data":"50f56a9f4a589a0d47cbaec5c8e176aa756c3680db66cc89ed9426f5bf803879"}
Sep 16 19:11:21.610288 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.610196    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"
Sep 16 19:11:21.610288 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.610286    2552 scope.go:117] "RemoveContainer" containerID="82e3e7de0ff89180631e77063250ed0a0f98ba96c6a4097f95b53cbae3d29182"
Sep 16 19:11:21.619591 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.619564    2552 scope.go:117] "RemoveContainer" containerID="9082392b6eeaa0507134595077cd8342ad6a2296fc79bba77191c0f275ea5357"
Sep 16 19:11:21.629038 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.629014    2552 scope.go:117] "RemoveContainer" containerID="c657bd8aaa8dc9fb63dbaa3115afe041edc5ccd42e03cb21239dde27324eb0fe"
Sep 16 19:11:21.640437 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.640402    2552 scope.go:117] "RemoveContainer" containerID="82e3e7de0ff89180631e77063250ed0a0f98ba96c6a4097f95b53cbae3d29182"
Sep 16 19:11:21.640833 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:11:21.640815    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e3e7de0ff89180631e77063250ed0a0f98ba96c6a4097f95b53cbae3d29182\": container with ID starting with 82e3e7de0ff89180631e77063250ed0a0f98ba96c6a4097f95b53cbae3d29182 not found: ID does not exist" containerID="82e3e7de0ff89180631e77063250ed0a0f98ba96c6a4097f95b53cbae3d29182"
Sep 16 19:11:21.640902 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.640842    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e3e7de0ff89180631e77063250ed0a0f98ba96c6a4097f95b53cbae3d29182"} err="failed to get container status \"82e3e7de0ff89180631e77063250ed0a0f98ba96c6a4097f95b53cbae3d29182\": rpc error: code = NotFound desc = could not find container \"82e3e7de0ff89180631e77063250ed0a0f98ba96c6a4097f95b53cbae3d29182\": container with ID starting with 82e3e7de0ff89180631e77063250ed0a0f98ba96c6a4097f95b53cbae3d29182 not found: ID does not exist"
Sep 16 19:11:21.640902 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.640860    2552 scope.go:117] "RemoveContainer" containerID="9082392b6eeaa0507134595077cd8342ad6a2296fc79bba77191c0f275ea5357"
Sep 16 19:11:21.641218 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:11:21.641188    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9082392b6eeaa0507134595077cd8342ad6a2296fc79bba77191c0f275ea5357\": container with ID starting with 9082392b6eeaa0507134595077cd8342ad6a2296fc79bba77191c0f275ea5357 not found: ID does not exist" containerID="9082392b6eeaa0507134595077cd8342ad6a2296fc79bba77191c0f275ea5357"
Sep 16 19:11:21.641357 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.641225    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9082392b6eeaa0507134595077cd8342ad6a2296fc79bba77191c0f275ea5357"} err="failed to get container status \"9082392b6eeaa0507134595077cd8342ad6a2296fc79bba77191c0f275ea5357\": rpc error: code = NotFound desc = could not find container \"9082392b6eeaa0507134595077cd8342ad6a2296fc79bba77191c0f275ea5357\": container with ID starting with 9082392b6eeaa0507134595077cd8342ad6a2296fc79bba77191c0f275ea5357 not found: ID does not exist"
Sep 16 19:11:21.641357 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.641254    2552 scope.go:117] "RemoveContainer" containerID="c657bd8aaa8dc9fb63dbaa3115afe041edc5ccd42e03cb21239dde27324eb0fe"
Sep 16 19:11:21.641920 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:11:21.641878    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c657bd8aaa8dc9fb63dbaa3115afe041edc5ccd42e03cb21239dde27324eb0fe\": container with ID starting with c657bd8aaa8dc9fb63dbaa3115afe041edc5ccd42e03cb21239dde27324eb0fe not found: ID does not exist" containerID="c657bd8aaa8dc9fb63dbaa3115afe041edc5ccd42e03cb21239dde27324eb0fe"
Sep 16 19:11:21.641999 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.641925    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c657bd8aaa8dc9fb63dbaa3115afe041edc5ccd42e03cb21239dde27324eb0fe"} err="failed to get container status \"c657bd8aaa8dc9fb63dbaa3115afe041edc5ccd42e03cb21239dde27324eb0fe\": rpc error: code = NotFound desc = could not find container \"c657bd8aaa8dc9fb63dbaa3115afe041edc5ccd42e03cb21239dde27324eb0fe\": container with ID starting with c657bd8aaa8dc9fb63dbaa3115afe041edc5ccd42e03cb21239dde27324eb0fe not found: ID does not exist"
Sep 16 19:11:21.651373 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.651330    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"]
Sep 16 19:11:21.661188 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:21.661153    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt"]
Sep 16 19:11:22.194217 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:22.194165    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:11:22.199375 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:22.199339    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: i/o timeout"
Sep 16 19:11:22.490820 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:22.490781    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:11:22.625359 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:22.625316    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f066541-420e-4f30-8be3-4cff06947d49" path="/var/lib/kubelet/pods/7f066541-420e-4f30-8be3-4cff06947d49/volumes"
Sep 16 19:11:25.633329 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:25.633291    2552 generic.go:356] "Generic (PLEG): container finished" podID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerID="0a8231552fbc75d5732b101e7d77e3bf8c6425a9696dc09123d0fa1f3077e05b" exitCode=0
Sep 16 19:11:25.633781 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:25.633365    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" event={"ID":"4f69e899-2c60-4d92-b39a-5f410344fd0a","Type":"ContainerDied","Data":"0a8231552fbc75d5732b101e7d77e3bf8c6425a9696dc09123d0fa1f3077e05b"}
Sep 16 19:11:26.641449 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:26.641368    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" event={"ID":"4f69e899-2c60-4d92-b39a-5f410344fd0a","Type":"ContainerStarted","Data":"3373d7ccde9e9336e786e7e466e20026ffb303a9146de7a5242b0dbec52dc46c"}
Sep 16 19:11:26.641449 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:26.641441    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" event={"ID":"4f69e899-2c60-4d92-b39a-5f410344fd0a","Type":"ContainerStarted","Data":"578f31fe7329982cacbe5b839be2a862c0a4990959b89b349368149afc312224"}
Sep 16 19:11:26.641870 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:26.641752    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:26.641943 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:26.641920    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:26.643435 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:26.643405    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:11:26.662919 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:26.662859    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" podStartSLOduration=2.933491994 podStartE2EDuration="6.66284081s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.729348816s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:26.66214401 +0000 UTC m=+1882.556331279" watchObservedRunningTime="2026-09-16 19:11:26.66284081 +0000 UTC m=+1882.557028104"
Sep 16 19:11:27.647232 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:27.647186    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:11:32.491294 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:32.491247    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:11:32.625184 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:32.625152    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:11:32.625352 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:11:32.625337    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:11:32.651797 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:32.651765    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:11:32.652453 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:32.652422    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:11:42.490621 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:42.490581    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:11:42.653225 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:42.653184    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:11:45.620075 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:45.620037    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:11:45.620513 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:11:45.620262    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:11:52.490860 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:52.490812    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:11:52.652633 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:52.652586    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:11:58.619662 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:11:58.619626    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:11:58.620132 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:11:58.619808    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:12:02.490454 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:02.490406    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:12:02.652957 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:02.652918    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:12:09.619596 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:09.619561    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:12:09.620264 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:12:09.619754    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:12:12.490526 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:12.490488    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:12:12.653058 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:12.653015    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:12:22.491951 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:22.491914    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:12:22.620736 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:22.620699    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:12:22.620926 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:12:22.620865    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:12:22.652638 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:22.652600    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:12:29.401770 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:29.401726    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"]
Sep 16 19:12:29.402232 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:29.401978    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kserve-container" containerID="cri-o://ed93c5fa48c89f0045d2a40e97a6ce350cc564e63099172ef512542f4d694060" gracePeriod=30
Sep 16 19:12:29.402232 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:29.402025    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kube-rbac-proxy" containerID="cri-o://4ec35421e14086b9a97eb45855a7fe39a7fc3261e991a3d4265aac4473e3d217" gracePeriod=30
Sep 16 19:12:29.936478 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:29.936432    2552 generic.go:356] "Generic (PLEG): container finished" podID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerID="4ec35421e14086b9a97eb45855a7fe39a7fc3261e991a3d4265aac4473e3d217" exitCode=2
Sep 16 19:12:29.936678 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:29.936506    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" event={"ID":"9b04d1fd-faf8-4b33-9c09-1d1da0244743","Type":"ContainerDied","Data":"4ec35421e14086b9a97eb45855a7fe39a7fc3261e991a3d4265aac4473e3d217"}
Sep 16 19:12:32.485886 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:32.485844    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 16 19:12:32.491400 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:32.491369    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:12:32.653252 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:32.653207    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:12:33.620281 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:33.620247    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:12:33.620681 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:12:33.620435    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:12:34.856638 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.856605    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:12:34.914403 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.914300    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kube-rbac-proxy-sar-config\") pod \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") "
Sep 16 19:12:34.914403 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.914369    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b04d1fd-faf8-4b33-9c09-1d1da0244743-proxy-tls\") pod \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") "
Sep 16 19:12:34.914639 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.914414    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b04d1fd-faf8-4b33-9c09-1d1da0244743-cabundle-cert\") pod \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") "
Sep 16 19:12:34.914639 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.914555    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cjrb\" (UniqueName: \"kubernetes.io/projected/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kube-api-access-7cjrb\") pod \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") "
Sep 16 19:12:34.914639 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.914593    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kserve-provision-location\") pod \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\" (UID: \"9b04d1fd-faf8-4b33-9c09-1d1da0244743\") "
Sep 16 19:12:34.914803 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.914777    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b04d1fd-faf8-4b33-9c09-1d1da0244743-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b04d1fd-faf8-4b33-9c09-1d1da0244743" (UID: "9b04d1fd-faf8-4b33-9c09-1d1da0244743"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:34.914853 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.914788    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b04d1fd-faf8-4b33-9c09-1d1da0244743" (UID: "9b04d1fd-faf8-4b33-9c09-1d1da0244743"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:34.914900 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.914866    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b04d1fd-faf8-4b33-9c09-1d1da0244743-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:12:34.914900 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.914882    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:12:34.915138 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.915105    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b04d1fd-faf8-4b33-9c09-1d1da0244743" (UID: "9b04d1fd-faf8-4b33-9c09-1d1da0244743"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:34.916591 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.916560    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b04d1fd-faf8-4b33-9c09-1d1da0244743-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b04d1fd-faf8-4b33-9c09-1d1da0244743" (UID: "9b04d1fd-faf8-4b33-9c09-1d1da0244743"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:34.916712 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.916688    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kube-api-access-7cjrb" (OuterVolumeSpecName: "kube-api-access-7cjrb") pod "9b04d1fd-faf8-4b33-9c09-1d1da0244743" (UID: "9b04d1fd-faf8-4b33-9c09-1d1da0244743"). InnerVolumeSpecName "kube-api-access-7cjrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:34.965776 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.965738    2552 generic.go:356] "Generic (PLEG): container finished" podID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerID="ed93c5fa48c89f0045d2a40e97a6ce350cc564e63099172ef512542f4d694060" exitCode=0
Sep 16 19:12:34.965960 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.965819    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"
Sep 16 19:12:34.965960 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.965822    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" event={"ID":"9b04d1fd-faf8-4b33-9c09-1d1da0244743","Type":"ContainerDied","Data":"ed93c5fa48c89f0045d2a40e97a6ce350cc564e63099172ef512542f4d694060"}
Sep 16 19:12:34.966118 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.965969    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj" event={"ID":"9b04d1fd-faf8-4b33-9c09-1d1da0244743","Type":"ContainerDied","Data":"0ea17a94c1b8c4d67b4efbc8f2a1be753a59ecff02047b95f304103f9c9003f9"}
Sep 16 19:12:34.966118 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.965999    2552 scope.go:117] "RemoveContainer" containerID="4ec35421e14086b9a97eb45855a7fe39a7fc3261e991a3d4265aac4473e3d217"
Sep 16 19:12:34.975329 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.975310    2552 scope.go:117] "RemoveContainer" containerID="ed93c5fa48c89f0045d2a40e97a6ce350cc564e63099172ef512542f4d694060"
Sep 16 19:12:34.983993 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.983969    2552 scope.go:117] "RemoveContainer" containerID="fb1ea8cc6b4afc09245092fd9151a02add0e330084ad98458d01ec76b5d0bbb1"
Sep 16 19:12:34.992308 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.992283    2552 scope.go:117] "RemoveContainer" containerID="4ec35421e14086b9a97eb45855a7fe39a7fc3261e991a3d4265aac4473e3d217"
Sep 16 19:12:34.992823 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:12:34.992776    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec35421e14086b9a97eb45855a7fe39a7fc3261e991a3d4265aac4473e3d217\": container with ID starting with 4ec35421e14086b9a97eb45855a7fe39a7fc3261e991a3d4265aac4473e3d217 not found: ID does not exist" containerID="4ec35421e14086b9a97eb45855a7fe39a7fc3261e991a3d4265aac4473e3d217"
Sep 16 19:12:34.992991 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.992827    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec35421e14086b9a97eb45855a7fe39a7fc3261e991a3d4265aac4473e3d217"} err="failed to get container status \"4ec35421e14086b9a97eb45855a7fe39a7fc3261e991a3d4265aac4473e3d217\": rpc error: code = NotFound desc = could not find container \"4ec35421e14086b9a97eb45855a7fe39a7fc3261e991a3d4265aac4473e3d217\": container with ID starting with 4ec35421e14086b9a97eb45855a7fe39a7fc3261e991a3d4265aac4473e3d217 not found: ID does not exist"
Sep 16 19:12:34.992991 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.992856    2552 scope.go:117] "RemoveContainer" containerID="ed93c5fa48c89f0045d2a40e97a6ce350cc564e63099172ef512542f4d694060"
Sep 16 19:12:34.993569 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:12:34.993542    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed93c5fa48c89f0045d2a40e97a6ce350cc564e63099172ef512542f4d694060\": container with ID starting with ed93c5fa48c89f0045d2a40e97a6ce350cc564e63099172ef512542f4d694060 not found: ID does not exist" containerID="ed93c5fa48c89f0045d2a40e97a6ce350cc564e63099172ef512542f4d694060"
Sep 16 19:12:34.993677 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.993575    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed93c5fa48c89f0045d2a40e97a6ce350cc564e63099172ef512542f4d694060"} err="failed to get container status \"ed93c5fa48c89f0045d2a40e97a6ce350cc564e63099172ef512542f4d694060\": rpc error: code = NotFound desc = could not find container \"ed93c5fa48c89f0045d2a40e97a6ce350cc564e63099172ef512542f4d694060\": container with ID starting with ed93c5fa48c89f0045d2a40e97a6ce350cc564e63099172ef512542f4d694060 not found: ID does not exist"
Sep 16 19:12:34.993677 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.993601    2552 scope.go:117] "RemoveContainer" containerID="fb1ea8cc6b4afc09245092fd9151a02add0e330084ad98458d01ec76b5d0bbb1"
Sep 16 19:12:34.993943 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:12:34.993923    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1ea8cc6b4afc09245092fd9151a02add0e330084ad98458d01ec76b5d0bbb1\": container with ID starting with fb1ea8cc6b4afc09245092fd9151a02add0e330084ad98458d01ec76b5d0bbb1 not found: ID does not exist" containerID="fb1ea8cc6b4afc09245092fd9151a02add0e330084ad98458d01ec76b5d0bbb1"
Sep 16 19:12:34.993999 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.993950    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1ea8cc6b4afc09245092fd9151a02add0e330084ad98458d01ec76b5d0bbb1"} err="failed to get container status \"fb1ea8cc6b4afc09245092fd9151a02add0e330084ad98458d01ec76b5d0bbb1\": rpc error: code = NotFound desc = could not find container \"fb1ea8cc6b4afc09245092fd9151a02add0e330084ad98458d01ec76b5d0bbb1\": container with ID starting with fb1ea8cc6b4afc09245092fd9151a02add0e330084ad98458d01ec76b5d0bbb1 not found: ID does not exist"
Sep 16 19:12:34.995194 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.995173    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"]
Sep 16 19:12:34.999574 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:34.999545    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj"]
Sep 16 19:12:35.016045 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:35.016015    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b04d1fd-faf8-4b33-9c09-1d1da0244743-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:12:35.016045 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:35.016045    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cjrb\" (UniqueName: \"kubernetes.io/projected/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kube-api-access-7cjrb\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:12:35.016239 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:35.016057    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b04d1fd-faf8-4b33-9c09-1d1da0244743-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:12:36.626137 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:36.625930    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" path="/var/lib/kubelet/pods/9b04d1fd-faf8-4b33-9c09-1d1da0244743/volumes"
Sep 16 19:12:42.652390 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:42.652346    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:12:44.625474 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:44.625444    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:12:44.625863 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:12:44.625634    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:12:52.653252 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:52.653221    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:12:59.619739 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:12:59.619701    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:12:59.620187 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:12:59.619873    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:13:00.846597 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:00.846555    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"]
Sep 16 19:13:00.847121 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:00.846827    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kserve-container" containerID="cri-o://578f31fe7329982cacbe5b839be2a862c0a4990959b89b349368149afc312224" gracePeriod=30
Sep 16 19:13:00.847121 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:00.846873    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kube-rbac-proxy" containerID="cri-o://3373d7ccde9e9336e786e7e466e20026ffb303a9146de7a5242b0dbec52dc46c" gracePeriod=30
Sep 16 19:13:01.055959 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.055912    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"]
Sep 16 19:13:01.056394 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056374    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kserve-container"
Sep 16 19:13:01.056491 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056396    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kserve-container"
Sep 16 19:13:01.056491 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056410    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="storage-initializer"
Sep 16 19:13:01.056491 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056418    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="storage-initializer"
Sep 16 19:13:01.056491 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056441    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kube-rbac-proxy"
Sep 16 19:13:01.056491 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056451    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kube-rbac-proxy"
Sep 16 19:13:01.056491 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056472    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kube-rbac-proxy"
Sep 16 19:13:01.056491 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056480    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kube-rbac-proxy"
Sep 16 19:13:01.056491 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056490    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="storage-initializer"
Sep 16 19:13:01.056891 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056500    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="storage-initializer"
Sep 16 19:13:01.056891 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056511    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kserve-container"
Sep 16 19:13:01.056891 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056520    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kserve-container"
Sep 16 19:13:01.056891 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056608    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kube-rbac-proxy"
Sep 16 19:13:01.056891 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056620    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kserve-container"
Sep 16 19:13:01.056891 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056630    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b04d1fd-faf8-4b33-9c09-1d1da0244743" containerName="kube-rbac-proxy"
Sep 16 19:13:01.056891 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.056641    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f066541-420e-4f30-8be3-4cff06947d49" containerName="kserve-container"
Sep 16 19:13:01.059884 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.059858    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.062197 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.062173    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:13:01.079545 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.079505    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"]
Sep 16 19:13:01.098317 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.098215    2552 generic.go:356] "Generic (PLEG): container finished" podID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerID="3373d7ccde9e9336e786e7e466e20026ffb303a9146de7a5242b0dbec52dc46c" exitCode=2
Sep 16 19:13:01.098317 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.098271    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" event={"ID":"4f69e899-2c60-4d92-b39a-5f410344fd0a","Type":"ContainerDied","Data":"3373d7ccde9e9336e786e7e466e20026ffb303a9146de7a5242b0dbec52dc46c"}
Sep 16 19:13:01.142971 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.142923    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51fb781b-1b88-4bb6-82e9-f03288a5a557-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-ktc84\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.143184 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.143034    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fb781b-1b88-4bb6-82e9-f03288a5a557-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-ktc84\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.143184 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.143055    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51fb781b-1b88-4bb6-82e9-f03288a5a557-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-ktc84\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.143184 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.143100    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51fb781b-1b88-4bb6-82e9-f03288a5a557-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-ktc84\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.143184 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.143174    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdss4\" (UniqueName: \"kubernetes.io/projected/51fb781b-1b88-4bb6-82e9-f03288a5a557-kube-api-access-mdss4\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-ktc84\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.244563 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.244513    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdss4\" (UniqueName: \"kubernetes.io/projected/51fb781b-1b88-4bb6-82e9-f03288a5a557-kube-api-access-mdss4\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-ktc84\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.244563 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.244563    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51fb781b-1b88-4bb6-82e9-f03288a5a557-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-ktc84\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.244822 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.244672    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fb781b-1b88-4bb6-82e9-f03288a5a557-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-ktc84\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.244822 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.244693    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51fb781b-1b88-4bb6-82e9-f03288a5a557-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-ktc84\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.244822 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.244714    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51fb781b-1b88-4bb6-82e9-f03288a5a557-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-ktc84\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.245178 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.245148    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51fb781b-1b88-4bb6-82e9-f03288a5a557-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-ktc84\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.245329 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.245310    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51fb781b-1b88-4bb6-82e9-f03288a5a557-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-ktc84\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.245421 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.245403    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51fb781b-1b88-4bb6-82e9-f03288a5a557-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-ktc84\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.247267 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.247244    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fb781b-1b88-4bb6-82e9-f03288a5a557-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-ktc84\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.255856 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.255827    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdss4\" (UniqueName: \"kubernetes.io/projected/51fb781b-1b88-4bb6-82e9-f03288a5a557-kube-api-access-mdss4\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-ktc84\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.370882 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.370774    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:01.653890 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:01.653862    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"]
Sep 16 19:13:01.656652 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:13:01.656618    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fb781b_1b88_4bb6_82e9_f03288a5a557.slice/crio-91d0f57a537ddf32f18fe12fa0e4173ceddc34dd1d18c537ab310473acd2aff2 WatchSource:0}: Error finding container 91d0f57a537ddf32f18fe12fa0e4173ceddc34dd1d18c537ab310473acd2aff2: Status 404 returned error can't find the container with id 91d0f57a537ddf32f18fe12fa0e4173ceddc34dd1d18c537ab310473acd2aff2
Sep 16 19:13:02.106193 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:02.106079    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" event={"ID":"51fb781b-1b88-4bb6-82e9-f03288a5a557","Type":"ContainerStarted","Data":"d7bc5c76679e8da28204dbff6766c9c28e2d7100d3b0db1442fb843329877c90"}
Sep 16 19:13:02.106193 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:02.106157    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" event={"ID":"51fb781b-1b88-4bb6-82e9-f03288a5a557","Type":"ContainerStarted","Data":"91d0f57a537ddf32f18fe12fa0e4173ceddc34dd1d18c537ab310473acd2aff2"}
Sep 16 19:13:02.647615 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:02.647567    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 16 19:13:02.653038 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:02.653004    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:13:06.128120 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.128061    2552 generic.go:356] "Generic (PLEG): container finished" podID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerID="578f31fe7329982cacbe5b839be2a862c0a4990959b89b349368149afc312224" exitCode=0
Sep 16 19:13:06.128572 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.128124    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" event={"ID":"4f69e899-2c60-4d92-b39a-5f410344fd0a","Type":"ContainerDied","Data":"578f31fe7329982cacbe5b839be2a862c0a4990959b89b349368149afc312224"}
Sep 16 19:13:06.130108 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.130059    2552 generic.go:356] "Generic (PLEG): container finished" podID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerID="d7bc5c76679e8da28204dbff6766c9c28e2d7100d3b0db1442fb843329877c90" exitCode=0
Sep 16 19:13:06.130236 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.130127    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" event={"ID":"51fb781b-1b88-4bb6-82e9-f03288a5a557","Type":"ContainerDied","Data":"d7bc5c76679e8da28204dbff6766c9c28e2d7100d3b0db1442fb843329877c90"}
Sep 16 19:13:06.224950 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.224916    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:13:06.293009 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.292982    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f69e899-2c60-4d92-b39a-5f410344fd0a-cabundle-cert\") pod \"4f69e899-2c60-4d92-b39a-5f410344fd0a\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") "
Sep 16 19:13:06.293183 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.293079    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvwrs\" (UniqueName: \"kubernetes.io/projected/4f69e899-2c60-4d92-b39a-5f410344fd0a-kube-api-access-tvwrs\") pod \"4f69e899-2c60-4d92-b39a-5f410344fd0a\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") "
Sep 16 19:13:06.293183 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.293164    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f69e899-2c60-4d92-b39a-5f410344fd0a-proxy-tls\") pod \"4f69e899-2c60-4d92-b39a-5f410344fd0a\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") "
Sep 16 19:13:06.293316 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.293193    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f69e899-2c60-4d92-b39a-5f410344fd0a-kube-rbac-proxy-sar-config\") pod \"4f69e899-2c60-4d92-b39a-5f410344fd0a\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") "
Sep 16 19:13:06.293546 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.293514    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f69e899-2c60-4d92-b39a-5f410344fd0a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f69e899-2c60-4d92-b39a-5f410344fd0a" (UID: "4f69e899-2c60-4d92-b39a-5f410344fd0a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:06.293647 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.293566    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f69e899-2c60-4d92-b39a-5f410344fd0a-kserve-provision-location\") pod \"4f69e899-2c60-4d92-b39a-5f410344fd0a\" (UID: \"4f69e899-2c60-4d92-b39a-5f410344fd0a\") "
Sep 16 19:13:06.293829 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.293533    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f69e899-2c60-4d92-b39a-5f410344fd0a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f69e899-2c60-4d92-b39a-5f410344fd0a" (UID: "4f69e899-2c60-4d92-b39a-5f410344fd0a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:06.293951 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.293829    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f69e899-2c60-4d92-b39a-5f410344fd0a-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:13:06.293951 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.293852    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f69e899-2c60-4d92-b39a-5f410344fd0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f69e899-2c60-4d92-b39a-5f410344fd0a" (UID: "4f69e899-2c60-4d92-b39a-5f410344fd0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:06.295494 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.295461    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f69e899-2c60-4d92-b39a-5f410344fd0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f69e899-2c60-4d92-b39a-5f410344fd0a" (UID: "4f69e899-2c60-4d92-b39a-5f410344fd0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:06.295603 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.295529    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f69e899-2c60-4d92-b39a-5f410344fd0a-kube-api-access-tvwrs" (OuterVolumeSpecName: "kube-api-access-tvwrs") pod "4f69e899-2c60-4d92-b39a-5f410344fd0a" (UID: "4f69e899-2c60-4d92-b39a-5f410344fd0a"). InnerVolumeSpecName "kube-api-access-tvwrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:06.394242 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.394208    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f69e899-2c60-4d92-b39a-5f410344fd0a-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:13:06.394242 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.394238    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f69e899-2c60-4d92-b39a-5f410344fd0a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:13:06.394242 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.394247    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f69e899-2c60-4d92-b39a-5f410344fd0a-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:13:06.394479 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:06.394256    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvwrs\" (UniqueName: \"kubernetes.io/projected/4f69e899-2c60-4d92-b39a-5f410344fd0a-kube-api-access-tvwrs\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:13:07.139318 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:07.139284    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"
Sep 16 19:13:07.139318 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:07.139290    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9" event={"ID":"4f69e899-2c60-4d92-b39a-5f410344fd0a","Type":"ContainerDied","Data":"f346f225ae29b60f10213b2d3e0c263fc522aeb9318601cfe7480486903108d1"}
Sep 16 19:13:07.139862 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:07.139363    2552 scope.go:117] "RemoveContainer" containerID="3373d7ccde9e9336e786e7e466e20026ffb303a9146de7a5242b0dbec52dc46c"
Sep 16 19:13:07.142230 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:07.142203    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" event={"ID":"51fb781b-1b88-4bb6-82e9-f03288a5a557","Type":"ContainerStarted","Data":"489740194f283118894a763b2190355742bd47b4176c4c875ec6dc3a937dc713"}
Sep 16 19:13:07.142349 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:07.142260    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" event={"ID":"51fb781b-1b88-4bb6-82e9-f03288a5a557","Type":"ContainerStarted","Data":"d219bd545b776895c8152487f26fd27afb16048c6a67bf8ff8f5d2fa2c7dbded"}
Sep 16 19:13:07.142539 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:07.142522    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:07.142604 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:07.142577    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:07.144396 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:07.144369    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:13:07.148465 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:07.148438    2552 scope.go:117] "RemoveContainer" containerID="578f31fe7329982cacbe5b839be2a862c0a4990959b89b349368149afc312224"
Sep 16 19:13:07.156808 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:07.156788    2552 scope.go:117] "RemoveContainer" containerID="0a8231552fbc75d5732b101e7d77e3bf8c6425a9696dc09123d0fa1f3077e05b"
Sep 16 19:13:07.169564 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:07.169507    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" podStartSLOduration=2.517378737 podStartE2EDuration="6.169491086s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.652112349s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:07.167134366 +0000 UTC m=+1983.061321635" watchObservedRunningTime="2026-09-16 19:13:07.169491086 +0000 UTC m=+1983.063678355"
Sep 16 19:13:07.187530 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:07.187495    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"]
Sep 16 19:13:07.191206 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:07.191174    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9"]
Sep 16 19:13:08.147877 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:08.147833    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:13:08.631017 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:08.630978    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" path="/var/lib/kubelet/pods/4f69e899-2c60-4d92-b39a-5f410344fd0a/volumes"
Sep 16 19:13:13.152425 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:13.152394    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:13:13.152916 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:13.152888    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:13:13.620124 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:13.620013    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:13:13.620362 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:13:13.620256    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:13:19.922050 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:19.922010    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"]
Sep 16 19:13:19.922460 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:19.922369    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kube-rbac-proxy"
Sep 16 19:13:19.922460 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:19.922382    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kube-rbac-proxy"
Sep 16 19:13:19.922460 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:19.922393    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="storage-initializer"
Sep 16 19:13:19.922460 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:19.922399    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="storage-initializer"
Sep 16 19:13:19.922460 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:19.922406    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kserve-container"
Sep 16 19:13:19.922460 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:19.922412    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kserve-container"
Sep 16 19:13:19.922460 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:19.922462    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kube-rbac-proxy"
Sep 16 19:13:19.922687 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:19.922475    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f69e899-2c60-4d92-b39a-5f410344fd0a" containerName="kserve-container"
Sep 16 19:13:19.927001 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:19.926975    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:19.929355 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:19.929327    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 16 19:13:19.929492 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:19.929418    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:13:19.938164 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:19.938133    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"]
Sep 16 19:13:20.008991 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.008946    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd2tw\" (UniqueName: \"kubernetes.io/projected/5c4eebf5-7235-44cf-88c7-91a7f321b424-kube-api-access-rd2tw\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.009218 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.009001    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c4eebf5-7235-44cf-88c7-91a7f321b424-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.009218 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.009077    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4eebf5-7235-44cf-88c7-91a7f321b424-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.009218 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.009173    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c4eebf5-7235-44cf-88c7-91a7f321b424-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.009333 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.009239    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c4eebf5-7235-44cf-88c7-91a7f321b424-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.110379 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.110337    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c4eebf5-7235-44cf-88c7-91a7f321b424-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.110379 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.110383    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd2tw\" (UniqueName: \"kubernetes.io/projected/5c4eebf5-7235-44cf-88c7-91a7f321b424-kube-api-access-rd2tw\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.110634 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.110408    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c4eebf5-7235-44cf-88c7-91a7f321b424-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.110634 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.110430    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4eebf5-7235-44cf-88c7-91a7f321b424-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.110634 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.110467    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c4eebf5-7235-44cf-88c7-91a7f321b424-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.110927 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.110900    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4eebf5-7235-44cf-88c7-91a7f321b424-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.111054 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.111033    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c4eebf5-7235-44cf-88c7-91a7f321b424-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.111147 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.111129    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c4eebf5-7235-44cf-88c7-91a7f321b424-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.113050 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.113026    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c4eebf5-7235-44cf-88c7-91a7f321b424-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.120695 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.120664    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd2tw\" (UniqueName: \"kubernetes.io/projected/5c4eebf5-7235-44cf-88c7-91a7f321b424-kube-api-access-rd2tw\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.240367 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.240269    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:20.522230 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:20.522141    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"]
Sep 16 19:13:20.525183 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:13:20.525143    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4eebf5_7235_44cf_88c7_91a7f321b424.slice/crio-b7e7e30a1a36023617fd8f8e3e9b1803e7f249089acf44e18183a5566567aa23 WatchSource:0}: Error finding container b7e7e30a1a36023617fd8f8e3e9b1803e7f249089acf44e18183a5566567aa23: Status 404 returned error can't find the container with id b7e7e30a1a36023617fd8f8e3e9b1803e7f249089acf44e18183a5566567aa23
Sep 16 19:13:21.207452 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:21.207405    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" event={"ID":"5c4eebf5-7235-44cf-88c7-91a7f321b424","Type":"ContainerStarted","Data":"86b6545cf925cff12ca48b814e99e5e4b9172cf83d18b7946d9ecdccd3334dcd"}
Sep 16 19:13:21.207926 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:21.207505    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" event={"ID":"5c4eebf5-7235-44cf-88c7-91a7f321b424","Type":"ContainerStarted","Data":"b7e7e30a1a36023617fd8f8e3e9b1803e7f249089acf44e18183a5566567aa23"}
Sep 16 19:13:23.153388 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:23.153340    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:13:26.232538 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:26.232500    2552 generic.go:356] "Generic (PLEG): container finished" podID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerID="86b6545cf925cff12ca48b814e99e5e4b9172cf83d18b7946d9ecdccd3334dcd" exitCode=0
Sep 16 19:13:26.232538 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:26.232544    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" event={"ID":"5c4eebf5-7235-44cf-88c7-91a7f321b424","Type":"ContainerDied","Data":"86b6545cf925cff12ca48b814e99e5e4b9172cf83d18b7946d9ecdccd3334dcd"}
Sep 16 19:13:28.620619 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:28.620585    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:13:28.621004 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:13:28.620781    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:13:30.257380 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:30.257341    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" event={"ID":"5c4eebf5-7235-44cf-88c7-91a7f321b424","Type":"ContainerStarted","Data":"7dd8d017f84d228263343ce37e317a75bc9096ba03cf6ef1b6956225ea67a2ec"}
Sep 16 19:13:30.257893 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:30.257436    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" event={"ID":"5c4eebf5-7235-44cf-88c7-91a7f321b424","Type":"ContainerStarted","Data":"0c08e049d87737e86330a2c1c72d63b5d3ddf00731464f40c635fb09ea7b631b"}
Sep 16 19:13:30.257893 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:30.257719    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:30.284171 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:30.284110    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" podStartSLOduration=2.682515646 podStartE2EDuration="11.284064663s" totalImagesPullingTime="3.827266861s" totalInitContainerRuntime="4.774282156s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:30.282241267 +0000 UTC m=+2006.176428536" watchObservedRunningTime="2026-09-16 19:13:30.284064663 +0000 UTC m=+2006.178251932"
Sep 16 19:13:31.262029 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:31.261960    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:31.263596 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:31.263559    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 19:13:32.266490 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:32.266447    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 19:13:33.153404 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:33.153359    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:13:37.271506 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:37.271470    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:37.272060 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:37.272030    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 19:13:42.620596 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:42.620561    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:13:43.153364 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:43.153314    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:13:44.323164 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:44.323127    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerStarted","Data":"a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"}
Sep 16 19:13:44.323568 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:44.323333    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:13:47.273062 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:47.273020    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:13:53.152908 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:13:53.152857    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:14:02.488229 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:02.488194    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"]
Sep 16 19:14:02.488800 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:02.488433    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kserve-container" containerID="cri-o://0c08e049d87737e86330a2c1c72d63b5d3ddf00731464f40c635fb09ea7b631b" gracePeriod=30
Sep 16 19:14:02.488800 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:02.488498    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kube-rbac-proxy" containerID="cri-o://7dd8d017f84d228263343ce37e317a75bc9096ba03cf6ef1b6956225ea67a2ec" gracePeriod=30
Sep 16 19:14:03.153282 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:03.153238    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:14:03.415973 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:03.415871    2552 generic.go:356] "Generic (PLEG): container finished" podID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerID="7dd8d017f84d228263343ce37e317a75bc9096ba03cf6ef1b6956225ea67a2ec" exitCode=2
Sep 16 19:14:03.415973 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:03.415939    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" event={"ID":"5c4eebf5-7235-44cf-88c7-91a7f321b424","Type":"ContainerDied","Data":"7dd8d017f84d228263343ce37e317a75bc9096ba03cf6ef1b6956225ea67a2ec"}
Sep 16 19:14:07.267591 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:07.267545    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 16 19:14:12.267390 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:12.267340    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 16 19:14:13.153528 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:13.153481    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:14:15.331063 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:15.331031    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:14:17.267594 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:17.267551    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 16 19:14:17.268017 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:17.267680    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:14:22.267637 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:22.267577    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 16 19:14:23.153422 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:23.153377    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:14:27.267593 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:27.267497    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 16 19:14:32.267278 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:32.267231    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 16 19:14:32.552489 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:32.552461    2552 generic.go:356] "Generic (PLEG): container finished" podID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerID="0c08e049d87737e86330a2c1c72d63b5d3ddf00731464f40c635fb09ea7b631b" exitCode=137
Sep 16 19:14:32.552598 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:32.552516    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" event={"ID":"5c4eebf5-7235-44cf-88c7-91a7f321b424","Type":"ContainerDied","Data":"0c08e049d87737e86330a2c1c72d63b5d3ddf00731464f40c635fb09ea7b631b"}
Sep 16 19:14:33.126860 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.126835    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:14:33.154001 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.153976    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:14:33.269714 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.269629    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c4eebf5-7235-44cf-88c7-91a7f321b424-kube-rbac-proxy-sar-config\") pod \"5c4eebf5-7235-44cf-88c7-91a7f321b424\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") "
Sep 16 19:14:33.269714 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.269666    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c4eebf5-7235-44cf-88c7-91a7f321b424-cabundle-cert\") pod \"5c4eebf5-7235-44cf-88c7-91a7f321b424\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") "
Sep 16 19:14:33.269714 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.269695    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd2tw\" (UniqueName: \"kubernetes.io/projected/5c4eebf5-7235-44cf-88c7-91a7f321b424-kube-api-access-rd2tw\") pod \"5c4eebf5-7235-44cf-88c7-91a7f321b424\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") "
Sep 16 19:14:33.270257 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.269724    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4eebf5-7235-44cf-88c7-91a7f321b424-kserve-provision-location\") pod \"5c4eebf5-7235-44cf-88c7-91a7f321b424\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") "
Sep 16 19:14:33.270257 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.269775    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c4eebf5-7235-44cf-88c7-91a7f321b424-proxy-tls\") pod \"5c4eebf5-7235-44cf-88c7-91a7f321b424\" (UID: \"5c4eebf5-7235-44cf-88c7-91a7f321b424\") "
Sep 16 19:14:33.270257 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.270005    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4eebf5-7235-44cf-88c7-91a7f321b424-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c4eebf5-7235-44cf-88c7-91a7f321b424" (UID: "5c4eebf5-7235-44cf-88c7-91a7f321b424"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:33.270257 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.270201    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4eebf5-7235-44cf-88c7-91a7f321b424-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c4eebf5-7235-44cf-88c7-91a7f321b424" (UID: "5c4eebf5-7235-44cf-88c7-91a7f321b424"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:33.271951 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.271925    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4eebf5-7235-44cf-88c7-91a7f321b424-kube-api-access-rd2tw" (OuterVolumeSpecName: "kube-api-access-rd2tw") pod "5c4eebf5-7235-44cf-88c7-91a7f321b424" (UID: "5c4eebf5-7235-44cf-88c7-91a7f321b424"). InnerVolumeSpecName "kube-api-access-rd2tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:33.272045 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.271986    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c4eebf5-7235-44cf-88c7-91a7f321b424-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c4eebf5-7235-44cf-88c7-91a7f321b424" (UID: "5c4eebf5-7235-44cf-88c7-91a7f321b424"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:33.280332 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.280309    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4eebf5-7235-44cf-88c7-91a7f321b424-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c4eebf5-7235-44cf-88c7-91a7f321b424" (UID: "5c4eebf5-7235-44cf-88c7-91a7f321b424"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:33.370608 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.370576    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4eebf5-7235-44cf-88c7-91a7f321b424-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:14:33.370608 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.370605    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c4eebf5-7235-44cf-88c7-91a7f321b424-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:14:33.370781 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.370618    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c4eebf5-7235-44cf-88c7-91a7f321b424-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:14:33.370781 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.370625    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c4eebf5-7235-44cf-88c7-91a7f321b424-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:14:33.370781 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.370637    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd2tw\" (UniqueName: \"kubernetes.io/projected/5c4eebf5-7235-44cf-88c7-91a7f321b424-kube-api-access-rd2tw\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:14:33.559303 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.559223    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"
Sep 16 19:14:33.559303 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.559243    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s" event={"ID":"5c4eebf5-7235-44cf-88c7-91a7f321b424","Type":"ContainerDied","Data":"b7e7e30a1a36023617fd8f8e3e9b1803e7f249089acf44e18183a5566567aa23"}
Sep 16 19:14:33.559303 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.559305    2552 scope.go:117] "RemoveContainer" containerID="7dd8d017f84d228263343ce37e317a75bc9096ba03cf6ef1b6956225ea67a2ec"
Sep 16 19:14:33.568068 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.568050    2552 scope.go:117] "RemoveContainer" containerID="0c08e049d87737e86330a2c1c72d63b5d3ddf00731464f40c635fb09ea7b631b"
Sep 16 19:14:33.575613 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.575597    2552 scope.go:117] "RemoveContainer" containerID="86b6545cf925cff12ca48b814e99e5e4b9172cf83d18b7946d9ecdccd3334dcd"
Sep 16 19:14:33.586642 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.586617    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"]
Sep 16 19:14:33.597245 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:33.597222    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s"]
Sep 16 19:14:34.625284 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:34.625251    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" path="/var/lib/kubelet/pods/5c4eebf5-7235-44cf-88c7-91a7f321b424/volumes"
Sep 16 19:14:41.097925 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:41.097894    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"]
Sep 16 19:14:41.098401 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:41.098141    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kserve-container" containerID="cri-o://d219bd545b776895c8152487f26fd27afb16048c6a67bf8ff8f5d2fa2c7dbded" gracePeriod=30
Sep 16 19:14:41.098401 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:41.098216    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kube-rbac-proxy" containerID="cri-o://489740194f283118894a763b2190355742bd47b4176c4c875ec6dc3a937dc713" gracePeriod=30
Sep 16 19:14:41.597331 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:41.597299    2552 generic.go:356] "Generic (PLEG): container finished" podID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerID="489740194f283118894a763b2190355742bd47b4176c4c875ec6dc3a937dc713" exitCode=2
Sep 16 19:14:41.597482 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:41.597354    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" event={"ID":"51fb781b-1b88-4bb6-82e9-f03288a5a557","Type":"ContainerDied","Data":"489740194f283118894a763b2190355742bd47b4176c4c875ec6dc3a937dc713"}
Sep 16 19:14:43.148270 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:43.148231    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 16 19:14:43.153488 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:43.153457    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:14:45.254630 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.254602    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:14:45.355941 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.355885    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdss4\" (UniqueName: \"kubernetes.io/projected/51fb781b-1b88-4bb6-82e9-f03288a5a557-kube-api-access-mdss4\") pod \"51fb781b-1b88-4bb6-82e9-f03288a5a557\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") "
Sep 16 19:14:45.355941 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.355933    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51fb781b-1b88-4bb6-82e9-f03288a5a557-cabundle-cert\") pod \"51fb781b-1b88-4bb6-82e9-f03288a5a557\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") "
Sep 16 19:14:45.356137 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.355953    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fb781b-1b88-4bb6-82e9-f03288a5a557-proxy-tls\") pod \"51fb781b-1b88-4bb6-82e9-f03288a5a557\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") "
Sep 16 19:14:45.356137 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.355978    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51fb781b-1b88-4bb6-82e9-f03288a5a557-kube-rbac-proxy-sar-config\") pod \"51fb781b-1b88-4bb6-82e9-f03288a5a557\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") "
Sep 16 19:14:45.356137 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.355996    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51fb781b-1b88-4bb6-82e9-f03288a5a557-kserve-provision-location\") pod \"51fb781b-1b88-4bb6-82e9-f03288a5a557\" (UID: \"51fb781b-1b88-4bb6-82e9-f03288a5a557\") "
Sep 16 19:14:45.356408 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.356381    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fb781b-1b88-4bb6-82e9-f03288a5a557-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51fb781b-1b88-4bb6-82e9-f03288a5a557" (UID: "51fb781b-1b88-4bb6-82e9-f03288a5a557"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:45.356408 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.356387    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fb781b-1b88-4bb6-82e9-f03288a5a557-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51fb781b-1b88-4bb6-82e9-f03288a5a557" (UID: "51fb781b-1b88-4bb6-82e9-f03288a5a557"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:45.356408 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.356399    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fb781b-1b88-4bb6-82e9-f03288a5a557-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51fb781b-1b88-4bb6-82e9-f03288a5a557" (UID: "51fb781b-1b88-4bb6-82e9-f03288a5a557"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:45.357948 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.357923    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51fb781b-1b88-4bb6-82e9-f03288a5a557-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51fb781b-1b88-4bb6-82e9-f03288a5a557" (UID: "51fb781b-1b88-4bb6-82e9-f03288a5a557"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:45.358045 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.357989    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fb781b-1b88-4bb6-82e9-f03288a5a557-kube-api-access-mdss4" (OuterVolumeSpecName: "kube-api-access-mdss4") pod "51fb781b-1b88-4bb6-82e9-f03288a5a557" (UID: "51fb781b-1b88-4bb6-82e9-f03288a5a557"). InnerVolumeSpecName "kube-api-access-mdss4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:45.456894 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.456872    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdss4\" (UniqueName: \"kubernetes.io/projected/51fb781b-1b88-4bb6-82e9-f03288a5a557-kube-api-access-mdss4\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:14:45.456894 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.456893    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51fb781b-1b88-4bb6-82e9-f03288a5a557-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:14:45.457024 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.456904    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fb781b-1b88-4bb6-82e9-f03288a5a557-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:14:45.457024 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.456912    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51fb781b-1b88-4bb6-82e9-f03288a5a557-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:14:45.457024 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.456920    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51fb781b-1b88-4bb6-82e9-f03288a5a557-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:14:45.622399 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.622368    2552 generic.go:356] "Generic (PLEG): container finished" podID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerID="d219bd545b776895c8152487f26fd27afb16048c6a67bf8ff8f5d2fa2c7dbded" exitCode=0
Sep 16 19:14:45.622527 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.622411    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"
Sep 16 19:14:45.622527 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.622441    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" event={"ID":"51fb781b-1b88-4bb6-82e9-f03288a5a557","Type":"ContainerDied","Data":"d219bd545b776895c8152487f26fd27afb16048c6a67bf8ff8f5d2fa2c7dbded"}
Sep 16 19:14:45.622527 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.622487    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84" event={"ID":"51fb781b-1b88-4bb6-82e9-f03288a5a557","Type":"ContainerDied","Data":"91d0f57a537ddf32f18fe12fa0e4173ceddc34dd1d18c537ab310473acd2aff2"}
Sep 16 19:14:45.622527 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.622516    2552 scope.go:117] "RemoveContainer" containerID="489740194f283118894a763b2190355742bd47b4176c4c875ec6dc3a937dc713"
Sep 16 19:14:45.631193 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.631178    2552 scope.go:117] "RemoveContainer" containerID="d219bd545b776895c8152487f26fd27afb16048c6a67bf8ff8f5d2fa2c7dbded"
Sep 16 19:14:45.638706 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.638691    2552 scope.go:117] "RemoveContainer" containerID="d7bc5c76679e8da28204dbff6766c9c28e2d7100d3b0db1442fb843329877c90"
Sep 16 19:14:45.645434 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.645419    2552 scope.go:117] "RemoveContainer" containerID="489740194f283118894a763b2190355742bd47b4176c4c875ec6dc3a937dc713"
Sep 16 19:14:45.645667 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:14:45.645650    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489740194f283118894a763b2190355742bd47b4176c4c875ec6dc3a937dc713\": container with ID starting with 489740194f283118894a763b2190355742bd47b4176c4c875ec6dc3a937dc713 not found: ID does not exist" containerID="489740194f283118894a763b2190355742bd47b4176c4c875ec6dc3a937dc713"
Sep 16 19:14:45.645717 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.645677    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489740194f283118894a763b2190355742bd47b4176c4c875ec6dc3a937dc713"} err="failed to get container status \"489740194f283118894a763b2190355742bd47b4176c4c875ec6dc3a937dc713\": rpc error: code = NotFound desc = could not find container \"489740194f283118894a763b2190355742bd47b4176c4c875ec6dc3a937dc713\": container with ID starting with 489740194f283118894a763b2190355742bd47b4176c4c875ec6dc3a937dc713 not found: ID does not exist"
Sep 16 19:14:45.645717 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.645694    2552 scope.go:117] "RemoveContainer" containerID="d219bd545b776895c8152487f26fd27afb16048c6a67bf8ff8f5d2fa2c7dbded"
Sep 16 19:14:45.645919 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:14:45.645905    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d219bd545b776895c8152487f26fd27afb16048c6a67bf8ff8f5d2fa2c7dbded\": container with ID starting with d219bd545b776895c8152487f26fd27afb16048c6a67bf8ff8f5d2fa2c7dbded not found: ID does not exist" containerID="d219bd545b776895c8152487f26fd27afb16048c6a67bf8ff8f5d2fa2c7dbded"
Sep 16 19:14:45.645959 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.645922    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d219bd545b776895c8152487f26fd27afb16048c6a67bf8ff8f5d2fa2c7dbded"} err="failed to get container status \"d219bd545b776895c8152487f26fd27afb16048c6a67bf8ff8f5d2fa2c7dbded\": rpc error: code = NotFound desc = could not find container \"d219bd545b776895c8152487f26fd27afb16048c6a67bf8ff8f5d2fa2c7dbded\": container with ID starting with d219bd545b776895c8152487f26fd27afb16048c6a67bf8ff8f5d2fa2c7dbded not found: ID does not exist"
Sep 16 19:14:45.645959 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.645933    2552 scope.go:117] "RemoveContainer" containerID="d7bc5c76679e8da28204dbff6766c9c28e2d7100d3b0db1442fb843329877c90"
Sep 16 19:14:45.646244 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:14:45.646222    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7bc5c76679e8da28204dbff6766c9c28e2d7100d3b0db1442fb843329877c90\": container with ID starting with d7bc5c76679e8da28204dbff6766c9c28e2d7100d3b0db1442fb843329877c90 not found: ID does not exist" containerID="d7bc5c76679e8da28204dbff6766c9c28e2d7100d3b0db1442fb843329877c90"
Sep 16 19:14:45.646303 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.646252    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7bc5c76679e8da28204dbff6766c9c28e2d7100d3b0db1442fb843329877c90"} err="failed to get container status \"d7bc5c76679e8da28204dbff6766c9c28e2d7100d3b0db1442fb843329877c90\": rpc error: code = NotFound desc = could not find container \"d7bc5c76679e8da28204dbff6766c9c28e2d7100d3b0db1442fb843329877c90\": container with ID starting with d7bc5c76679e8da28204dbff6766c9c28e2d7100d3b0db1442fb843329877c90 not found: ID does not exist"
Sep 16 19:14:45.649180 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.649077    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"]
Sep 16 19:14:45.654850 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:45.654828    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84"]
Sep 16 19:14:46.329916 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.329883    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"]
Sep 16 19:14:46.330375 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330228    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="storage-initializer"
Sep 16 19:14:46.330375 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330242    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="storage-initializer"
Sep 16 19:14:46.330375 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330251    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kserve-container"
Sep 16 19:14:46.330375 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330256    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kserve-container"
Sep 16 19:14:46.330375 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330263    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="storage-initializer"
Sep 16 19:14:46.330375 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330271    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="storage-initializer"
Sep 16 19:14:46.330375 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330289    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kube-rbac-proxy"
Sep 16 19:14:46.330375 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330294    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kube-rbac-proxy"
Sep 16 19:14:46.330375 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330305    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kube-rbac-proxy"
Sep 16 19:14:46.330375 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330310    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kube-rbac-proxy"
Sep 16 19:14:46.330375 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330325    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kserve-container"
Sep 16 19:14:46.330375 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330330    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kserve-container"
Sep 16 19:14:46.330375 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330383    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kube-rbac-proxy"
Sep 16 19:14:46.330775 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330391    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kserve-container"
Sep 16 19:14:46.330775 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330398    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" containerName="kube-rbac-proxy"
Sep 16 19:14:46.330775 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.330405    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c4eebf5-7235-44cf-88c7-91a7f321b424" containerName="kserve-container"
Sep 16 19:14:46.334814 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.334796    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.337143 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.337121    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:14:46.337264 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.337123    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:14:46.337264 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.337186    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 16 19:14:46.338117 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.338079    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:14:46.338222 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.338133    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:14:46.338222 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.338147    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 19:14:46.338222 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.338163    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:14:46.348228 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.348205    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"]
Sep 16 19:14:46.363369 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.363347    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b326b42-437a-4c6f-97f3-73957ae419c4-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.363464 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.363378    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57rvg\" (UniqueName: \"kubernetes.io/projected/5b326b42-437a-4c6f-97f3-73957ae419c4-kube-api-access-57rvg\") pod \"isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.363464 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.363405    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b326b42-437a-4c6f-97f3-73957ae419c4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.363464 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.363431    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b326b42-437a-4c6f-97f3-73957ae419c4-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.363596 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.363476    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b326b42-437a-4c6f-97f3-73957ae419c4-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.393413 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.393363    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"]
Sep 16 19:14:46.396751 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.396733    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:46.399057 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.399040    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 19:14:46.410030 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.410007    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"]
Sep 16 19:14:46.464817 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.464792    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8tqw\" (UniqueName: \"kubernetes.io/projected/28fa0010-de20-4aad-a8bc-c37051c58d87-kube-api-access-v8tqw\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:46.464916 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.464834    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b326b42-437a-4c6f-97f3-73957ae419c4-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.464916 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.464871    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57rvg\" (UniqueName: \"kubernetes.io/projected/5b326b42-437a-4c6f-97f3-73957ae419c4-kube-api-access-57rvg\") pod \"isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.464916 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.464897    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28fa0010-de20-4aad-a8bc-c37051c58d87-proxy-tls\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:46.465054 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.464926    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28fa0010-de20-4aad-a8bc-c37051c58d87-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:46.465054 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.464956    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b326b42-437a-4c6f-97f3-73957ae419c4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.465054 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.464987    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b326b42-437a-4c6f-97f3-73957ae419c4-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.465054 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.465027    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28fa0010-de20-4aad-a8bc-c37051c58d87-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:46.465265 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.465067    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b326b42-437a-4c6f-97f3-73957ae419c4-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.465265 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.465114    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28fa0010-de20-4aad-a8bc-c37051c58d87-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:46.465493 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.465470    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b326b42-437a-4c6f-97f3-73957ae419c4-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.465651 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.465620    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b326b42-437a-4c6f-97f3-73957ae419c4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.465651 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.465637    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b326b42-437a-4c6f-97f3-73957ae419c4-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.467468 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.467452    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b326b42-437a-4c6f-97f3-73957ae419c4-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.484576 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.484555    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57rvg\" (UniqueName: \"kubernetes.io/projected/5b326b42-437a-4c6f-97f3-73957ae419c4-kube-api-access-57rvg\") pod \"isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.565769 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.565740    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28fa0010-de20-4aad-a8bc-c37051c58d87-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:46.565896 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.565779    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28fa0010-de20-4aad-a8bc-c37051c58d87-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:46.565896 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.565818    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8tqw\" (UniqueName: \"kubernetes.io/projected/28fa0010-de20-4aad-a8bc-c37051c58d87-kube-api-access-v8tqw\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:46.565896 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.565841    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28fa0010-de20-4aad-a8bc-c37051c58d87-proxy-tls\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:46.565896 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.565866    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28fa0010-de20-4aad-a8bc-c37051c58d87-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:46.566163 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:14:46.565984    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:14:46.566163 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:14:46.566050    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28fa0010-de20-4aad-a8bc-c37051c58d87-proxy-tls podName:28fa0010-de20-4aad-a8bc-c37051c58d87 nodeName:}" failed. No retries permitted until 2026-09-16 19:14:47.066034432 +0000 UTC m=+2082.960221678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/28fa0010-de20-4aad-a8bc-c37051c58d87-proxy-tls") pod "isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" (UID: "28fa0010-de20-4aad-a8bc-c37051c58d87") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:14:46.566259 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.566234    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28fa0010-de20-4aad-a8bc-c37051c58d87-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:46.566456 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.566439    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28fa0010-de20-4aad-a8bc-c37051c58d87-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:46.566493 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.566440    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28fa0010-de20-4aad-a8bc-c37051c58d87-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:46.580221 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.580167    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8tqw\" (UniqueName: \"kubernetes.io/projected/28fa0010-de20-4aad-a8bc-c37051c58d87-kube-api-access-v8tqw\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:46.625367 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.625341    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51fb781b-1b88-4bb6-82e9-f03288a5a557" path="/var/lib/kubelet/pods/51fb781b-1b88-4bb6-82e9-f03288a5a557/volumes"
Sep 16 19:14:46.645109 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.645069    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:46.919780 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:46.919757    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"]
Sep 16 19:14:46.922026 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:14:46.921989    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b326b42_437a_4c6f_97f3_73957ae419c4.slice/crio-d16e2c2d8cf0eda3c361a953b8c0a22a362b54d36362b38accd33a568fbe6f60 WatchSource:0}: Error finding container d16e2c2d8cf0eda3c361a953b8c0a22a362b54d36362b38accd33a568fbe6f60: Status 404 returned error can't find the container with id d16e2c2d8cf0eda3c361a953b8c0a22a362b54d36362b38accd33a568fbe6f60
Sep 16 19:14:47.070657 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:47.070629    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28fa0010-de20-4aad-a8bc-c37051c58d87-proxy-tls\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:47.072828 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:47.072804    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28fa0010-de20-4aad-a8bc-c37051c58d87-proxy-tls\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:47.306023 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:47.305931    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:47.595550 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:47.595522    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"]
Sep 16 19:14:47.597918 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:14:47.597887    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fa0010_de20_4aad_a8bc_c37051c58d87.slice/crio-47ebae0c15ab9de57a9a38132ebc8041af1792817d9ea3c4b8d6054a7bc136d4 WatchSource:0}: Error finding container 47ebae0c15ab9de57a9a38132ebc8041af1792817d9ea3c4b8d6054a7bc136d4: Status 404 returned error can't find the container with id 47ebae0c15ab9de57a9a38132ebc8041af1792817d9ea3c4b8d6054a7bc136d4
Sep 16 19:14:47.634334 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:47.634300    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" event={"ID":"5b326b42-437a-4c6f-97f3-73957ae419c4","Type":"ContainerStarted","Data":"a18712f9c185b78a9a3b9828c6edd48c7284b76a75682e6cca04809af111b2c0"}
Sep 16 19:14:47.634474 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:47.634372    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" event={"ID":"5b326b42-437a-4c6f-97f3-73957ae419c4","Type":"ContainerStarted","Data":"d16e2c2d8cf0eda3c361a953b8c0a22a362b54d36362b38accd33a568fbe6f60"}
Sep 16 19:14:47.635674 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:47.635649    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" event={"ID":"28fa0010-de20-4aad-a8bc-c37051c58d87","Type":"ContainerStarted","Data":"47ebae0c15ab9de57a9a38132ebc8041af1792817d9ea3c4b8d6054a7bc136d4"}
Sep 16 19:14:48.641155 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:48.641111    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" event={"ID":"28fa0010-de20-4aad-a8bc-c37051c58d87","Type":"ContainerStarted","Data":"ff2c2e822b246232f66adfae99775a26f5c88168c5e1aa34ce9a2e57cd9e7960"}
Sep 16 19:14:51.656894 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:51.656861    2552 generic.go:356] "Generic (PLEG): container finished" podID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerID="a18712f9c185b78a9a3b9828c6edd48c7284b76a75682e6cca04809af111b2c0" exitCode=0
Sep 16 19:14:51.657300 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:51.656940    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" event={"ID":"5b326b42-437a-4c6f-97f3-73957ae419c4","Type":"ContainerDied","Data":"a18712f9c185b78a9a3b9828c6edd48c7284b76a75682e6cca04809af111b2c0"}
Sep 16 19:14:51.658605 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:51.658582    2552 generic.go:356] "Generic (PLEG): container finished" podID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerID="ff2c2e822b246232f66adfae99775a26f5c88168c5e1aa34ce9a2e57cd9e7960" exitCode=0
Sep 16 19:14:51.658696 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:51.658661    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" event={"ID":"28fa0010-de20-4aad-a8bc-c37051c58d87","Type":"ContainerDied","Data":"ff2c2e822b246232f66adfae99775a26f5c88168c5e1aa34ce9a2e57cd9e7960"}
Sep 16 19:14:52.665677 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:52.665641    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" event={"ID":"5b326b42-437a-4c6f-97f3-73957ae419c4","Type":"ContainerStarted","Data":"f5f4ef71717d5f0de1ef8cafd4cb767827ed7977f0400d691698649d114ed8be"}
Sep 16 19:14:52.666147 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:52.665706    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" event={"ID":"5b326b42-437a-4c6f-97f3-73957ae419c4","Type":"ContainerStarted","Data":"5fe8e97652ab4bee5181ad7240fa7bb3a7ea85e508ffda3bdec54ddcc4e1a2e3"}
Sep 16 19:14:52.666147 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:52.666006    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:52.666147 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:52.666075    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:52.667926 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:52.667886    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:14:52.668055 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:52.667990    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" event={"ID":"28fa0010-de20-4aad-a8bc-c37051c58d87","Type":"ContainerStarted","Data":"c013a27134ca73938e8bd57e72dd93dc49e86e509741b1d1d54ef5c8066f7cbf"}
Sep 16 19:14:52.668055 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:52.668035    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" event={"ID":"28fa0010-de20-4aad-a8bc-c37051c58d87","Type":"ContainerStarted","Data":"78b57091f51ed4a679dd4f4a1943d7c5273278b5be1e079b7d4aabb769e75485"}
Sep 16 19:14:52.668310 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:52.668292    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:52.668399 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:52.668331    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:52.669372 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:52.669351    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:14:52.694349 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:52.694307    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" podStartSLOduration=2.885227693 podStartE2EDuration="6.694294167s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.809066474s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:52.693025807 +0000 UTC m=+2088.587213076" watchObservedRunningTime="2026-09-16 19:14:52.694294167 +0000 UTC m=+2088.588481497"
Sep 16 19:14:52.721919 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:52.721832    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" podStartSLOduration=2.827825056 podStartE2EDuration="6.721810566s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.89398551s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:52.717434287 +0000 UTC m=+2088.611621553" watchObservedRunningTime="2026-09-16 19:14:52.721810566 +0000 UTC m=+2088.615997835"
Sep 16 19:14:53.672614 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:53.672556    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:14:53.673036 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:53.672662    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:14:58.676965 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:58.676936    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:14:58.677406 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:58.677290    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:14:58.677464 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:58.677425    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:14:58.677711 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:14:58.677688    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:15:08.677473 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:08.677437    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:15:08.677880 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:08.677610    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:15:18.678345 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:18.678309    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:15:18.678812 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:18.678313    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:15:28.677422 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:28.677383    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:15:28.677909 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:28.677860    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:15:38.678264 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:38.678229    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:15:38.678730 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:38.678228    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:15:48.678298 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:48.678258    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:15:48.678745 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:48.678258    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:15:58.677518 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:58.677481    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:15:58.679758 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:58.677774    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:15:59.970464 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:59.970436    2552 generic.go:356] "Generic (PLEG): container finished" podID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308" exitCode=1
Sep 16 19:15:59.970792 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:59.970512    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerDied","Data":"a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"}
Sep 16 19:15:59.970792 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:59.970574    2552 scope.go:117] "RemoveContainer" containerID="eb1b8fac26a89ac2e25ef0e8999a352b59ebb82e6ed7311b8dcbcd24fb7ecbdc"
Sep 16 19:15:59.970910 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:15:59.970844    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:15:59.971029 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:15:59.971009    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:16:00.328496 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:00.328411    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:16:00.976819 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:00.976793    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:16:00.977212 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:16:00.976960    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:16:01.620074 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:01.620035    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:16:06.788738 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:06.788704    2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:16:06.789128 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:06.789022    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:16:06.789233 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:16:06.789213    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:16:08.678221 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:08.678191    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:16:08.853391 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:08.853352    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"]
Sep 16 19:16:08.853702 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:08.853665    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kserve-container" containerID="cri-o://78b57091f51ed4a679dd4f4a1943d7c5273278b5be1e079b7d4aabb769e75485" gracePeriod=30
Sep 16 19:16:08.853810 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:08.853746    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kube-rbac-proxy" containerID="cri-o://c013a27134ca73938e8bd57e72dd93dc49e86e509741b1d1d54ef5c8066f7cbf" gracePeriod=30
Sep 16 19:16:08.854421 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:08.854277    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:16:09.015449 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:09.015419    2552 generic.go:356] "Generic (PLEG): container finished" podID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerID="c013a27134ca73938e8bd57e72dd93dc49e86e509741b1d1d54ef5c8066f7cbf" exitCode=2
Sep 16 19:16:09.015586 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:09.015453    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" event={"ID":"28fa0010-de20-4aad-a8bc-c37051c58d87","Type":"ContainerDied","Data":"c013a27134ca73938e8bd57e72dd93dc49e86e509741b1d1d54ef5c8066f7cbf"}
Sep 16 19:16:13.003562 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.003541    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:16:13.036495 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.036466    2552 generic.go:356] "Generic (PLEG): container finished" podID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerID="78b57091f51ed4a679dd4f4a1943d7c5273278b5be1e079b7d4aabb769e75485" exitCode=0
Sep 16 19:16:13.036643 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.036522    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"
Sep 16 19:16:13.036643 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.036546    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" event={"ID":"28fa0010-de20-4aad-a8bc-c37051c58d87","Type":"ContainerDied","Data":"78b57091f51ed4a679dd4f4a1943d7c5273278b5be1e079b7d4aabb769e75485"}
Sep 16 19:16:13.036643 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.036620    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt" event={"ID":"28fa0010-de20-4aad-a8bc-c37051c58d87","Type":"ContainerDied","Data":"47ebae0c15ab9de57a9a38132ebc8041af1792817d9ea3c4b8d6054a7bc136d4"}
Sep 16 19:16:13.036808 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.036661    2552 scope.go:117] "RemoveContainer" containerID="c013a27134ca73938e8bd57e72dd93dc49e86e509741b1d1d54ef5c8066f7cbf"
Sep 16 19:16:13.044827 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.044805    2552 scope.go:117] "RemoveContainer" containerID="78b57091f51ed4a679dd4f4a1943d7c5273278b5be1e079b7d4aabb769e75485"
Sep 16 19:16:13.053414 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.053393    2552 scope.go:117] "RemoveContainer" containerID="ff2c2e822b246232f66adfae99775a26f5c88168c5e1aa34ce9a2e57cd9e7960"
Sep 16 19:16:13.060918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.060900    2552 scope.go:117] "RemoveContainer" containerID="c013a27134ca73938e8bd57e72dd93dc49e86e509741b1d1d54ef5c8066f7cbf"
Sep 16 19:16:13.061212 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:16:13.061186    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c013a27134ca73938e8bd57e72dd93dc49e86e509741b1d1d54ef5c8066f7cbf\": container with ID starting with c013a27134ca73938e8bd57e72dd93dc49e86e509741b1d1d54ef5c8066f7cbf not found: ID does not exist" containerID="c013a27134ca73938e8bd57e72dd93dc49e86e509741b1d1d54ef5c8066f7cbf"
Sep 16 19:16:13.061266 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.061220    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c013a27134ca73938e8bd57e72dd93dc49e86e509741b1d1d54ef5c8066f7cbf"} err="failed to get container status \"c013a27134ca73938e8bd57e72dd93dc49e86e509741b1d1d54ef5c8066f7cbf\": rpc error: code = NotFound desc = could not find container \"c013a27134ca73938e8bd57e72dd93dc49e86e509741b1d1d54ef5c8066f7cbf\": container with ID starting with c013a27134ca73938e8bd57e72dd93dc49e86e509741b1d1d54ef5c8066f7cbf not found: ID does not exist"
Sep 16 19:16:13.061266 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.061239    2552 scope.go:117] "RemoveContainer" containerID="78b57091f51ed4a679dd4f4a1943d7c5273278b5be1e079b7d4aabb769e75485"
Sep 16 19:16:13.062023 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:16:13.061896    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b57091f51ed4a679dd4f4a1943d7c5273278b5be1e079b7d4aabb769e75485\": container with ID starting with 78b57091f51ed4a679dd4f4a1943d7c5273278b5be1e079b7d4aabb769e75485 not found: ID does not exist" containerID="78b57091f51ed4a679dd4f4a1943d7c5273278b5be1e079b7d4aabb769e75485"
Sep 16 19:16:13.062023 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.061929    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b57091f51ed4a679dd4f4a1943d7c5273278b5be1e079b7d4aabb769e75485"} err="failed to get container status \"78b57091f51ed4a679dd4f4a1943d7c5273278b5be1e079b7d4aabb769e75485\": rpc error: code = NotFound desc = could not find container \"78b57091f51ed4a679dd4f4a1943d7c5273278b5be1e079b7d4aabb769e75485\": container with ID starting with 78b57091f51ed4a679dd4f4a1943d7c5273278b5be1e079b7d4aabb769e75485 not found: ID does not exist"
Sep 16 19:16:13.062023 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.061951    2552 scope.go:117] "RemoveContainer" containerID="ff2c2e822b246232f66adfae99775a26f5c88168c5e1aa34ce9a2e57cd9e7960"
Sep 16 19:16:13.062600 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:16:13.062549    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2c2e822b246232f66adfae99775a26f5c88168c5e1aa34ce9a2e57cd9e7960\": container with ID starting with ff2c2e822b246232f66adfae99775a26f5c88168c5e1aa34ce9a2e57cd9e7960 not found: ID does not exist" containerID="ff2c2e822b246232f66adfae99775a26f5c88168c5e1aa34ce9a2e57cd9e7960"
Sep 16 19:16:13.062600 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.062578    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2c2e822b246232f66adfae99775a26f5c88168c5e1aa34ce9a2e57cd9e7960"} err="failed to get container status \"ff2c2e822b246232f66adfae99775a26f5c88168c5e1aa34ce9a2e57cd9e7960\": rpc error: code = NotFound desc = could not find container \"ff2c2e822b246232f66adfae99775a26f5c88168c5e1aa34ce9a2e57cd9e7960\": container with ID starting with ff2c2e822b246232f66adfae99775a26f5c88168c5e1aa34ce9a2e57cd9e7960 not found: ID does not exist"
Sep 16 19:16:13.063614 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.063529    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8tqw\" (UniqueName: \"kubernetes.io/projected/28fa0010-de20-4aad-a8bc-c37051c58d87-kube-api-access-v8tqw\") pod \"28fa0010-de20-4aad-a8bc-c37051c58d87\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") "
Sep 16 19:16:13.063614 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.063597    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28fa0010-de20-4aad-a8bc-c37051c58d87-cabundle-cert\") pod \"28fa0010-de20-4aad-a8bc-c37051c58d87\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") "
Sep 16 19:16:13.063775 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.063653    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28fa0010-de20-4aad-a8bc-c37051c58d87-proxy-tls\") pod \"28fa0010-de20-4aad-a8bc-c37051c58d87\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") "
Sep 16 19:16:13.063775 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.063693    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28fa0010-de20-4aad-a8bc-c37051c58d87-kserve-provision-location\") pod \"28fa0010-de20-4aad-a8bc-c37051c58d87\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") "
Sep 16 19:16:13.063775 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.063732    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28fa0010-de20-4aad-a8bc-c37051c58d87-kube-rbac-proxy-sar-config\") pod \"28fa0010-de20-4aad-a8bc-c37051c58d87\" (UID: \"28fa0010-de20-4aad-a8bc-c37051c58d87\") "
Sep 16 19:16:13.064343 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.064234    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fa0010-de20-4aad-a8bc-c37051c58d87-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "28fa0010-de20-4aad-a8bc-c37051c58d87" (UID: "28fa0010-de20-4aad-a8bc-c37051c58d87"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:13.064446 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.064373    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fa0010-de20-4aad-a8bc-c37051c58d87-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "28fa0010-de20-4aad-a8bc-c37051c58d87" (UID: "28fa0010-de20-4aad-a8bc-c37051c58d87"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:13.064908 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.064881    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fa0010-de20-4aad-a8bc-c37051c58d87-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "28fa0010-de20-4aad-a8bc-c37051c58d87" (UID: "28fa0010-de20-4aad-a8bc-c37051c58d87"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:13.066286 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.066259    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28fa0010-de20-4aad-a8bc-c37051c58d87-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "28fa0010-de20-4aad-a8bc-c37051c58d87" (UID: "28fa0010-de20-4aad-a8bc-c37051c58d87"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:13.066769 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.066747    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fa0010-de20-4aad-a8bc-c37051c58d87-kube-api-access-v8tqw" (OuterVolumeSpecName: "kube-api-access-v8tqw") pod "28fa0010-de20-4aad-a8bc-c37051c58d87" (UID: "28fa0010-de20-4aad-a8bc-c37051c58d87"). InnerVolumeSpecName "kube-api-access-v8tqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:13.165215 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.165192    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28fa0010-de20-4aad-a8bc-c37051c58d87-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:16:13.165215 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.165213    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8tqw\" (UniqueName: \"kubernetes.io/projected/28fa0010-de20-4aad-a8bc-c37051c58d87-kube-api-access-v8tqw\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:16:13.165354 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.165222    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28fa0010-de20-4aad-a8bc-c37051c58d87-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:16:13.165354 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.165230    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28fa0010-de20-4aad-a8bc-c37051c58d87-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:16:13.165354 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.165238    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28fa0010-de20-4aad-a8bc-c37051c58d87-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:16:13.361020 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.360994    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"]
Sep 16 19:16:13.367577 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:13.367555    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt"]
Sep 16 19:16:14.624768 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:14.624727    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" path="/var/lib/kubelet/pods/28fa0010-de20-4aad-a8bc-c37051c58d87/volumes"
Sep 16 19:16:21.619643 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:21.619601    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:16:21.620162 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:16:21.619792    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:16:26.330790 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.330754    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"]
Sep 16 19:16:26.331220 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.330989    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kserve-container" containerID="cri-o://5fe8e97652ab4bee5181ad7240fa7bb3a7ea85e508ffda3bdec54ddcc4e1a2e3" gracePeriod=30
Sep 16 19:16:26.331220 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.331030    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kube-rbac-proxy" containerID="cri-o://f5f4ef71717d5f0de1ef8cafd4cb767827ed7977f0400d691698649d114ed8be" gracePeriod=30
Sep 16 19:16:26.508493 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.508464    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"]
Sep 16 19:16:26.508815 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.508801    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kserve-container"
Sep 16 19:16:26.508870 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.508817    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kserve-container"
Sep 16 19:16:26.508870 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.508834    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kube-rbac-proxy"
Sep 16 19:16:26.508870 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.508839    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kube-rbac-proxy"
Sep 16 19:16:26.508870 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.508858    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="storage-initializer"
Sep 16 19:16:26.508870 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.508866    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="storage-initializer"
Sep 16 19:16:26.509046 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.508918    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kube-rbac-proxy"
Sep 16 19:16:26.509046 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.508931    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="28fa0010-de20-4aad-a8bc-c37051c58d87" containerName="kserve-container"
Sep 16 19:16:26.511973 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.511956    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.514425 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.514406    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 19:16:26.523741 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.523719    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"]
Sep 16 19:16:26.662777 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.662740    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2nx7\" (UniqueName: \"kubernetes.io/projected/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kube-api-access-t2nx7\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.662953 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.662811    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.662953 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.662884    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-proxy-tls\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.662953 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.662944    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.663065 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.663034    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.763873 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.763842    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.764040 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.763897    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-proxy-tls\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.764040 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.763930    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.764040 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.763973    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.764040 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.764011    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2nx7\" (UniqueName: \"kubernetes.io/projected/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kube-api-access-t2nx7\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.764331 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.764305    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.764601 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.764582    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.764649 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.764596    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.766357 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.766337    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-proxy-tls\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.774023 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.773994    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2nx7\" (UniqueName: \"kubernetes.io/projected/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kube-api-access-t2nx7\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:26.821643 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:26.821619    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:27.096068 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:27.096038    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"]
Sep 16 19:16:27.098463 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:16:27.098439    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac21032f_39dd_41b3_9a1a_6d1f8ae41652.slice/crio-ba5d8a1a286510b3aec09a1eec9d6a24445f856dcf76ebb23f7ba40dd3e81aae WatchSource:0}: Error finding container ba5d8a1a286510b3aec09a1eec9d6a24445f856dcf76ebb23f7ba40dd3e81aae: Status 404 returned error can't find the container with id ba5d8a1a286510b3aec09a1eec9d6a24445f856dcf76ebb23f7ba40dd3e81aae
Sep 16 19:16:27.104461 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:27.104437    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" event={"ID":"ac21032f-39dd-41b3-9a1a-6d1f8ae41652","Type":"ContainerStarted","Data":"ba5d8a1a286510b3aec09a1eec9d6a24445f856dcf76ebb23f7ba40dd3e81aae"}
Sep 16 19:16:27.106668 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:27.106644    2552 generic.go:356] "Generic (PLEG): container finished" podID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerID="f5f4ef71717d5f0de1ef8cafd4cb767827ed7977f0400d691698649d114ed8be" exitCode=2
Sep 16 19:16:27.106800 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:27.106716    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" event={"ID":"5b326b42-437a-4c6f-97f3-73957ae419c4","Type":"ContainerDied","Data":"f5f4ef71717d5f0de1ef8cafd4cb767827ed7977f0400d691698649d114ed8be"}
Sep 16 19:16:28.112146 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:28.112114    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" event={"ID":"ac21032f-39dd-41b3-9a1a-6d1f8ae41652","Type":"ContainerStarted","Data":"9b1282e9fa1901fc8349d13b7125f7cc3b6e333c001d0f3014df10f0ee3f1341"}
Sep 16 19:16:28.673396 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:28.673361    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 16 19:16:28.677701 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:28.677668    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:16:30.670079 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.670057    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:16:30.799488 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.799414    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b326b42-437a-4c6f-97f3-73957ae419c4-cabundle-cert\") pod \"5b326b42-437a-4c6f-97f3-73957ae419c4\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") "
Sep 16 19:16:30.799488 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.799459    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b326b42-437a-4c6f-97f3-73957ae419c4-kserve-provision-location\") pod \"5b326b42-437a-4c6f-97f3-73957ae419c4\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") "
Sep 16 19:16:30.799672 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.799498    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b326b42-437a-4c6f-97f3-73957ae419c4-proxy-tls\") pod \"5b326b42-437a-4c6f-97f3-73957ae419c4\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") "
Sep 16 19:16:30.799672 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.799534    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b326b42-437a-4c6f-97f3-73957ae419c4-kube-rbac-proxy-sar-config\") pod \"5b326b42-437a-4c6f-97f3-73957ae419c4\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") "
Sep 16 19:16:30.799672 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.799558    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57rvg\" (UniqueName: \"kubernetes.io/projected/5b326b42-437a-4c6f-97f3-73957ae419c4-kube-api-access-57rvg\") pod \"5b326b42-437a-4c6f-97f3-73957ae419c4\" (UID: \"5b326b42-437a-4c6f-97f3-73957ae419c4\") "
Sep 16 19:16:30.799896 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.799861    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b326b42-437a-4c6f-97f3-73957ae419c4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5b326b42-437a-4c6f-97f3-73957ae419c4" (UID: "5b326b42-437a-4c6f-97f3-73957ae419c4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:30.799896 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.799864    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b326b42-437a-4c6f-97f3-73957ae419c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b326b42-437a-4c6f-97f3-73957ae419c4" (UID: "5b326b42-437a-4c6f-97f3-73957ae419c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:30.800065 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.799967    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b326b42-437a-4c6f-97f3-73957ae419c4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b326b42-437a-4c6f-97f3-73957ae419c4" (UID: "5b326b42-437a-4c6f-97f3-73957ae419c4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:30.801697 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.801668    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b326b42-437a-4c6f-97f3-73957ae419c4-kube-api-access-57rvg" (OuterVolumeSpecName: "kube-api-access-57rvg") pod "5b326b42-437a-4c6f-97f3-73957ae419c4" (UID: "5b326b42-437a-4c6f-97f3-73957ae419c4"). InnerVolumeSpecName "kube-api-access-57rvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:30.801697 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.801685    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b326b42-437a-4c6f-97f3-73957ae419c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b326b42-437a-4c6f-97f3-73957ae419c4" (UID: "5b326b42-437a-4c6f-97f3-73957ae419c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:30.900881 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.900855    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b326b42-437a-4c6f-97f3-73957ae419c4-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:16:30.900881 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.900876    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b326b42-437a-4c6f-97f3-73957ae419c4-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:16:30.901038 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.900887    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b326b42-437a-4c6f-97f3-73957ae419c4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:16:30.901038 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.900895    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57rvg\" (UniqueName: \"kubernetes.io/projected/5b326b42-437a-4c6f-97f3-73957ae419c4-kube-api-access-57rvg\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:16:30.901038 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:30.900904    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b326b42-437a-4c6f-97f3-73957ae419c4-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:16:31.126417 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.126385    2552 generic.go:356] "Generic (PLEG): container finished" podID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerID="9b1282e9fa1901fc8349d13b7125f7cc3b6e333c001d0f3014df10f0ee3f1341" exitCode=0
Sep 16 19:16:31.126531 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.126459    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" event={"ID":"ac21032f-39dd-41b3-9a1a-6d1f8ae41652","Type":"ContainerDied","Data":"9b1282e9fa1901fc8349d13b7125f7cc3b6e333c001d0f3014df10f0ee3f1341"}
Sep 16 19:16:31.128501 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.128479    2552 generic.go:356] "Generic (PLEG): container finished" podID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerID="5fe8e97652ab4bee5181ad7240fa7bb3a7ea85e508ffda3bdec54ddcc4e1a2e3" exitCode=0
Sep 16 19:16:31.128593 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.128524    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" event={"ID":"5b326b42-437a-4c6f-97f3-73957ae419c4","Type":"ContainerDied","Data":"5fe8e97652ab4bee5181ad7240fa7bb3a7ea85e508ffda3bdec54ddcc4e1a2e3"}
Sep 16 19:16:31.128593 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.128547    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"
Sep 16 19:16:31.128593 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.128583    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b" event={"ID":"5b326b42-437a-4c6f-97f3-73957ae419c4","Type":"ContainerDied","Data":"d16e2c2d8cf0eda3c361a953b8c0a22a362b54d36362b38accd33a568fbe6f60"}
Sep 16 19:16:31.128756 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.128622    2552 scope.go:117] "RemoveContainer" containerID="f5f4ef71717d5f0de1ef8cafd4cb767827ed7977f0400d691698649d114ed8be"
Sep 16 19:16:31.139166 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.139147    2552 scope.go:117] "RemoveContainer" containerID="5fe8e97652ab4bee5181ad7240fa7bb3a7ea85e508ffda3bdec54ddcc4e1a2e3"
Sep 16 19:16:31.148478 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.148454    2552 scope.go:117] "RemoveContainer" containerID="a18712f9c185b78a9a3b9828c6edd48c7284b76a75682e6cca04809af111b2c0"
Sep 16 19:16:31.163214 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.163193    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"]
Sep 16 19:16:31.167655 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.167629    2552 scope.go:117] "RemoveContainer" containerID="f5f4ef71717d5f0de1ef8cafd4cb767827ed7977f0400d691698649d114ed8be"
Sep 16 19:16:31.167928 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:16:31.167900    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f4ef71717d5f0de1ef8cafd4cb767827ed7977f0400d691698649d114ed8be\": container with ID starting with f5f4ef71717d5f0de1ef8cafd4cb767827ed7977f0400d691698649d114ed8be not found: ID does not exist" containerID="f5f4ef71717d5f0de1ef8cafd4cb767827ed7977f0400d691698649d114ed8be"
Sep 16 19:16:31.168000 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.167930    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f4ef71717d5f0de1ef8cafd4cb767827ed7977f0400d691698649d114ed8be"} err="failed to get container status \"f5f4ef71717d5f0de1ef8cafd4cb767827ed7977f0400d691698649d114ed8be\": rpc error: code = NotFound desc = could not find container \"f5f4ef71717d5f0de1ef8cafd4cb767827ed7977f0400d691698649d114ed8be\": container with ID starting with f5f4ef71717d5f0de1ef8cafd4cb767827ed7977f0400d691698649d114ed8be not found: ID does not exist"
Sep 16 19:16:31.168000 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.167953    2552 scope.go:117] "RemoveContainer" containerID="5fe8e97652ab4bee5181ad7240fa7bb3a7ea85e508ffda3bdec54ddcc4e1a2e3"
Sep 16 19:16:31.168291 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:16:31.168274    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe8e97652ab4bee5181ad7240fa7bb3a7ea85e508ffda3bdec54ddcc4e1a2e3\": container with ID starting with 5fe8e97652ab4bee5181ad7240fa7bb3a7ea85e508ffda3bdec54ddcc4e1a2e3 not found: ID does not exist" containerID="5fe8e97652ab4bee5181ad7240fa7bb3a7ea85e508ffda3bdec54ddcc4e1a2e3"
Sep 16 19:16:31.168339 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.168298    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe8e97652ab4bee5181ad7240fa7bb3a7ea85e508ffda3bdec54ddcc4e1a2e3"} err="failed to get container status \"5fe8e97652ab4bee5181ad7240fa7bb3a7ea85e508ffda3bdec54ddcc4e1a2e3\": rpc error: code = NotFound desc = could not find container \"5fe8e97652ab4bee5181ad7240fa7bb3a7ea85e508ffda3bdec54ddcc4e1a2e3\": container with ID starting with 5fe8e97652ab4bee5181ad7240fa7bb3a7ea85e508ffda3bdec54ddcc4e1a2e3 not found: ID does not exist"
Sep 16 19:16:31.168339 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.168314    2552 scope.go:117] "RemoveContainer" containerID="a18712f9c185b78a9a3b9828c6edd48c7284b76a75682e6cca04809af111b2c0"
Sep 16 19:16:31.168579 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:16:31.168557    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18712f9c185b78a9a3b9828c6edd48c7284b76a75682e6cca04809af111b2c0\": container with ID starting with a18712f9c185b78a9a3b9828c6edd48c7284b76a75682e6cca04809af111b2c0 not found: ID does not exist" containerID="a18712f9c185b78a9a3b9828c6edd48c7284b76a75682e6cca04809af111b2c0"
Sep 16 19:16:31.168618 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.168590    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18712f9c185b78a9a3b9828c6edd48c7284b76a75682e6cca04809af111b2c0"} err="failed to get container status \"a18712f9c185b78a9a3b9828c6edd48c7284b76a75682e6cca04809af111b2c0\": rpc error: code = NotFound desc = could not find container \"a18712f9c185b78a9a3b9828c6edd48c7284b76a75682e6cca04809af111b2c0\": container with ID starting with a18712f9c185b78a9a3b9828c6edd48c7284b76a75682e6cca04809af111b2c0 not found: ID does not exist"
Sep 16 19:16:31.172017 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:31.171996    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b"]
Sep 16 19:16:32.135746 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:32.135713    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" event={"ID":"ac21032f-39dd-41b3-9a1a-6d1f8ae41652","Type":"ContainerStarted","Data":"479412dad3acf6dc36fcb10fdad6270eb710e1e77d77f8ce18d708c3fb10dd10"}
Sep 16 19:16:32.136279 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:32.135781    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" event={"ID":"ac21032f-39dd-41b3-9a1a-6d1f8ae41652","Type":"ContainerStarted","Data":"801e97281ca3cdefa85714c03fa4d1110e70c22d983ffea82f90f8c860f8ddca"}
Sep 16 19:16:32.136279 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:32.135979    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:32.161014 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:32.160969    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" podStartSLOduration=2.314703099 podStartE2EDuration="6.160955239s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.84625214s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:32.158997251 +0000 UTC m=+2188.053184532" watchObservedRunningTime="2026-09-16 19:16:32.160955239 +0000 UTC m=+2188.055142582"
Sep 16 19:16:32.624647 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:32.624621    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" path="/var/lib/kubelet/pods/5b326b42-437a-4c6f-97f3-73957ae419c4/volumes"
Sep 16 19:16:33.141552 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:33.141525    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:33.142676 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:33.142649    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:16:34.145479 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:34.145438    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:16:34.621919 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:34.621891    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:16:34.622153 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:16:34.622129    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:16:39.150132 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:39.150081    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:16:39.150644 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:39.150614    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:16:49.151166 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:49.151127    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:16:49.620436 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:49.620362    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:16:49.620579 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:16:49.620526    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:16:59.150635 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:16:59.150596    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:17:02.620207 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:02.620171    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:17:02.620644 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:17:02.620388    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:17:09.150598 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:09.150558    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:17:17.619673 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:17.619642    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:17:17.620274 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:17:17.619799    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:17:19.151187 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:19.151150    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:17:29.150770 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:29.150692    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:17:32.620515 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:32.620484    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:17:32.620965 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:17:32.620647    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:17:39.151740 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:39.151706    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:17:43.620360 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:43.620333    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:17:43.620734 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:17:43.620476    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:17:46.505530 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:46.505490    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"]
Sep 16 19:17:46.506010 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:46.505737    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kserve-container" containerID="cri-o://801e97281ca3cdefa85714c03fa4d1110e70c22d983ffea82f90f8c860f8ddca" gracePeriod=30
Sep 16 19:17:46.506010 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:46.505804    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kube-rbac-proxy" containerID="cri-o://479412dad3acf6dc36fcb10fdad6270eb710e1e77d77f8ce18d708c3fb10dd10" gracePeriod=30
Sep 16 19:17:47.459855 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:47.459822    2552 generic.go:356] "Generic (PLEG): container finished" podID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerID="479412dad3acf6dc36fcb10fdad6270eb710e1e77d77f8ce18d708c3fb10dd10" exitCode=2
Sep 16 19:17:47.460039 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:47.459893    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" event={"ID":"ac21032f-39dd-41b3-9a1a-6d1f8ae41652","Type":"ContainerDied","Data":"479412dad3acf6dc36fcb10fdad6270eb710e1e77d77f8ce18d708c3fb10dd10"}
Sep 16 19:17:49.146195 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:49.146150    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Sep 16 19:17:49.151330 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:49.151292    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:17:50.475821 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.475786    2552 generic.go:356] "Generic (PLEG): container finished" podID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerID="801e97281ca3cdefa85714c03fa4d1110e70c22d983ffea82f90f8c860f8ddca" exitCode=0
Sep 16 19:17:50.476127 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.475844    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" event={"ID":"ac21032f-39dd-41b3-9a1a-6d1f8ae41652","Type":"ContainerDied","Data":"801e97281ca3cdefa85714c03fa4d1110e70c22d983ffea82f90f8c860f8ddca"}
Sep 16 19:17:50.547814 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.547793    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:17:50.682441 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.682420    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kube-rbac-proxy-sar-config\") pod \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") "
Sep 16 19:17:50.682573 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.682450    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-proxy-tls\") pod \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") "
Sep 16 19:17:50.682573 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.682468    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2nx7\" (UniqueName: \"kubernetes.io/projected/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kube-api-access-t2nx7\") pod \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") "
Sep 16 19:17:50.682573 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.682548    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kserve-provision-location\") pod \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") "
Sep 16 19:17:50.682701 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.682578    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-cabundle-cert\") pod \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\" (UID: \"ac21032f-39dd-41b3-9a1a-6d1f8ae41652\") "
Sep 16 19:17:50.682891 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.682804    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ac21032f-39dd-41b3-9a1a-6d1f8ae41652" (UID: "ac21032f-39dd-41b3-9a1a-6d1f8ae41652"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:50.682964 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.682824    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ac21032f-39dd-41b3-9a1a-6d1f8ae41652" (UID: "ac21032f-39dd-41b3-9a1a-6d1f8ae41652"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:50.682964 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.682952    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ac21032f-39dd-41b3-9a1a-6d1f8ae41652" (UID: "ac21032f-39dd-41b3-9a1a-6d1f8ae41652"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:50.684414 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.684398    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kube-api-access-t2nx7" (OuterVolumeSpecName: "kube-api-access-t2nx7") pod "ac21032f-39dd-41b3-9a1a-6d1f8ae41652" (UID: "ac21032f-39dd-41b3-9a1a-6d1f8ae41652"). InnerVolumeSpecName "kube-api-access-t2nx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:50.684465 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.684447    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ac21032f-39dd-41b3-9a1a-6d1f8ae41652" (UID: "ac21032f-39dd-41b3-9a1a-6d1f8ae41652"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:50.784051 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.784028    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:17:50.784051 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.784049    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:17:50.784193 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.784059    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:17:50.784193 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.784067    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:17:50.784193 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:50.784076    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2nx7\" (UniqueName: \"kubernetes.io/projected/ac21032f-39dd-41b3-9a1a-6d1f8ae41652-kube-api-access-t2nx7\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.482462 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.482428    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw" event={"ID":"ac21032f-39dd-41b3-9a1a-6d1f8ae41652","Type":"ContainerDied","Data":"ba5d8a1a286510b3aec09a1eec9d6a24445f856dcf76ebb23f7ba40dd3e81aae"}
Sep 16 19:17:51.482462 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.482449    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"
Sep 16 19:17:51.482936 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.482502    2552 scope.go:117] "RemoveContainer" containerID="479412dad3acf6dc36fcb10fdad6270eb710e1e77d77f8ce18d708c3fb10dd10"
Sep 16 19:17:51.490983 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.490966    2552 scope.go:117] "RemoveContainer" containerID="801e97281ca3cdefa85714c03fa4d1110e70c22d983ffea82f90f8c860f8ddca"
Sep 16 19:17:51.498031 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.498003    2552 scope.go:117] "RemoveContainer" containerID="9b1282e9fa1901fc8349d13b7125f7cc3b6e333c001d0f3014df10f0ee3f1341"
Sep 16 19:17:51.505746 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.505727    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"]
Sep 16 19:17:51.510237 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.510219    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw"]
Sep 16 19:17:51.781743 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.781651    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"]
Sep 16 19:17:51.781954 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.781942    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kube-rbac-proxy"
Sep 16 19:17:51.781998 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.781956    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kube-rbac-proxy"
Sep 16 19:17:51.781998 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.781971    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="storage-initializer"
Sep 16 19:17:51.781998 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.781977    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="storage-initializer"
Sep 16 19:17:51.781998 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.781986    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kserve-container"
Sep 16 19:17:51.781998 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.781992    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kserve-container"
Sep 16 19:17:51.782179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.782003    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kube-rbac-proxy"
Sep 16 19:17:51.782179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.782008    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kube-rbac-proxy"
Sep 16 19:17:51.782179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.782016    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kserve-container"
Sep 16 19:17:51.782179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.782021    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kserve-container"
Sep 16 19:17:51.782179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.782027    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="storage-initializer"
Sep 16 19:17:51.782179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.782033    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="storage-initializer"
Sep 16 19:17:51.782179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.782100    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kube-rbac-proxy"
Sep 16 19:17:51.782179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.782109    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kube-rbac-proxy"
Sep 16 19:17:51.782179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.782124    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b326b42-437a-4c6f-97f3-73957ae419c4" containerName="kserve-container"
Sep 16 19:17:51.782179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.782130    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" containerName="kserve-container"
Sep 16 19:17:51.786472 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.786456    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:51.788866 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.788847    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 19:17:51.788967 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.788854    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:17:51.788967 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.788931    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:17:51.789293 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.789279    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:17:51.789714 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.789700    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:17:51.789927 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.789911    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 19:17:51.789970 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.789957    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:17:51.798775 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.798752    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"]
Sep 16 19:17:51.890986 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.890950    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-proxy-tls\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:51.890986 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.890990    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8lj6\" (UniqueName: \"kubernetes.io/projected/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kube-api-access-p8lj6\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:51.891178 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.891009    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:51.891178 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.891036    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:51.891178 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.891147    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:51.992063 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.992036    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-proxy-tls\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:51.992063 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.992064    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8lj6\" (UniqueName: \"kubernetes.io/projected/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kube-api-access-p8lj6\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:51.992306 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.992080    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:51.992306 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.992142    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:51.992306 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:17:51.992182    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 16 19:17:51.992306 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.992214    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:51.992306 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:17:51.992242    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-proxy-tls podName:d8627bb9-53de-4aad-9a02-ca265ff5a3c3 nodeName:}" failed. No retries permitted until 2026-09-16 19:17:52.492228313 +0000 UTC m=+2268.386415579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-proxy-tls") pod "isvc-canary-stop-predictor-69f76fb68d-zjjqs" (UID: "d8627bb9-53de-4aad-9a02-ca265ff5a3c3") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 16 19:17:51.992552 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.992532    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:51.992797 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.992783    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:51.992835 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:51.992794    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:52.001627 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:52.001603    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8lj6\" (UniqueName: \"kubernetes.io/projected/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kube-api-access-p8lj6\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:52.496346 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:52.496318    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-proxy-tls\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:52.498653 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:52.498626    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-proxy-tls\") pod \"isvc-canary-stop-predictor-69f76fb68d-zjjqs\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:52.625026 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:52.625000    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac21032f-39dd-41b3-9a1a-6d1f8ae41652" path="/var/lib/kubelet/pods/ac21032f-39dd-41b3-9a1a-6d1f8ae41652/volumes"
Sep 16 19:17:52.696389 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:52.696365    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:52.970764 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:52.970737    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"]
Sep 16 19:17:52.973332 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:17:52.973305    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8627bb9_53de_4aad_9a02_ca265ff5a3c3.slice/crio-62e4107eed16fdd7015140926064552f966a22f5a18a73280d3048e0ff2174a5 WatchSource:0}: Error finding container 62e4107eed16fdd7015140926064552f966a22f5a18a73280d3048e0ff2174a5: Status 404 returned error can't find the container with id 62e4107eed16fdd7015140926064552f966a22f5a18a73280d3048e0ff2174a5
Sep 16 19:17:53.493944 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:53.493911    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" event={"ID":"d8627bb9-53de-4aad-9a02-ca265ff5a3c3","Type":"ContainerStarted","Data":"bbf481a9b1156ccf00b122596dbdc702b3e3cf2e8d2fd0f22caed1ab34d2223f"}
Sep 16 19:17:53.494250 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:53.493980    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" event={"ID":"d8627bb9-53de-4aad-9a02-ca265ff5a3c3","Type":"ContainerStarted","Data":"62e4107eed16fdd7015140926064552f966a22f5a18a73280d3048e0ff2174a5"}
Sep 16 19:17:54.621917 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:54.621885    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:17:54.622308 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:17:54.622107    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:17:57.511865 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:57.511826    2552 generic.go:356] "Generic (PLEG): container finished" podID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerID="bbf481a9b1156ccf00b122596dbdc702b3e3cf2e8d2fd0f22caed1ab34d2223f" exitCode=0
Sep 16 19:17:57.512317 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:57.511895    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" event={"ID":"d8627bb9-53de-4aad-9a02-ca265ff5a3c3","Type":"ContainerDied","Data":"bbf481a9b1156ccf00b122596dbdc702b3e3cf2e8d2fd0f22caed1ab34d2223f"}
Sep 16 19:17:58.518002 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:58.517971    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" event={"ID":"d8627bb9-53de-4aad-9a02-ca265ff5a3c3","Type":"ContainerStarted","Data":"fda739ce5d3e4ee1be9fe314e5d75652d3ea3aa496ac82e4594e7af7ece68398"}
Sep 16 19:17:58.518389 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:58.518033    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" event={"ID":"d8627bb9-53de-4aad-9a02-ca265ff5a3c3","Type":"ContainerStarted","Data":"491cad9d7e4c4aaf80c88dc5866e1a60ce61eaa50083ab2a580c0ccb94f1b11e"}
Sep 16 19:17:58.518389 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:58.518325    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:58.518529 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:58.518509    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:17:58.519574 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:58.519552    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:17:58.539391 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:58.539346    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" podStartSLOduration=3.620649879 podStartE2EDuration="7.539333247s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.918683368s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:58.537473571 +0000 UTC m=+2274.431660839" watchObservedRunningTime="2026-09-16 19:17:58.539333247 +0000 UTC m=+2274.433520515"
Sep 16 19:17:59.522202 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:17:59.522163    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:18:04.527892 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:04.527628    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:18:04.528379 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:04.528264    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:18:08.630483 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:08.626649    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:18:08.630483 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:18:08.626902    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:18:14.528632 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:14.528592    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:18:20.551350 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.551315    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"]
Sep 16 19:18:20.555971 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.555951    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:20.559192 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.559164    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 19:18:20.559192 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.559163    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 16 19:18:20.566192 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.566167    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"]
Sep 16 19:18:20.608099 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.608059    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96524\" (UniqueName: \"kubernetes.io/projected/49c6c8fb-505f-4207-817c-8171b84d8f58-kube-api-access-96524\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:20.608214 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.608151    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49c6c8fb-505f-4207-817c-8171b84d8f58-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:20.608268 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.608214    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c6c8fb-505f-4207-817c-8171b84d8f58-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:20.608315 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.608266    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49c6c8fb-505f-4207-817c-8171b84d8f58-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:20.608315 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.608286    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49c6c8fb-505f-4207-817c-8171b84d8f58-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:20.709133 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.709073    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49c6c8fb-505f-4207-817c-8171b84d8f58-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:20.709291 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.709142    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49c6c8fb-505f-4207-817c-8171b84d8f58-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:20.709291 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.709185    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96524\" (UniqueName: \"kubernetes.io/projected/49c6c8fb-505f-4207-817c-8171b84d8f58-kube-api-access-96524\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:20.709291 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.709231    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49c6c8fb-505f-4207-817c-8171b84d8f58-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:20.709291 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.709272    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c6c8fb-505f-4207-817c-8171b84d8f58-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:20.709509 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:18:20.709418    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 16 19:18:20.709509 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:18:20.709504    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49c6c8fb-505f-4207-817c-8171b84d8f58-proxy-tls podName:49c6c8fb-505f-4207-817c-8171b84d8f58 nodeName:}" failed. No retries permitted until 2026-09-16 19:18:21.209485652 +0000 UTC m=+2297.103672905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/49c6c8fb-505f-4207-817c-8171b84d8f58-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d" (UID: "49c6c8fb-505f-4207-817c-8171b84d8f58") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 16 19:18:20.709624 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.709563    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49c6c8fb-505f-4207-817c-8171b84d8f58-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:20.709736 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.709716    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49c6c8fb-505f-4207-817c-8171b84d8f58-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:20.709857 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.709842    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49c6c8fb-505f-4207-817c-8171b84d8f58-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:20.718265 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:20.718241    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96524\" (UniqueName: \"kubernetes.io/projected/49c6c8fb-505f-4207-817c-8171b84d8f58-kube-api-access-96524\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:21.212500 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:21.212465    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c6c8fb-505f-4207-817c-8171b84d8f58-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:21.215415 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:21.215381    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c6c8fb-505f-4207-817c-8171b84d8f58-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:21.466341 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:21.466241    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:21.743381 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:21.743355    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"]
Sep 16 19:18:21.745586 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:18:21.745559    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c6c8fb_505f_4207_817c_8171b84d8f58.slice/crio-cb7d2e6ebae4dbf5e9c70cdf602905f291e3a7a6c5db9da6d72fb967498c376d WatchSource:0}: Error finding container cb7d2e6ebae4dbf5e9c70cdf602905f291e3a7a6c5db9da6d72fb967498c376d: Status 404 returned error can't find the container with id cb7d2e6ebae4dbf5e9c70cdf602905f291e3a7a6c5db9da6d72fb967498c376d
Sep 16 19:18:22.627695 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:22.627660    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d" event={"ID":"49c6c8fb-505f-4207-817c-8171b84d8f58","Type":"ContainerStarted","Data":"b5c4b656f031376efdf3300fd81c92415a2cf3c54f0fb0f9dad00b17522c1f5c"}
Sep 16 19:18:22.627920 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:22.627726    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d" event={"ID":"49c6c8fb-505f-4207-817c-8171b84d8f58","Type":"ContainerStarted","Data":"cb7d2e6ebae4dbf5e9c70cdf602905f291e3a7a6c5db9da6d72fb967498c376d"}
Sep 16 19:18:23.620501 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:23.620471    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:18:23.620967 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:18:23.620653    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:18:24.528539 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:24.528501    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:18:26.647070 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:26.647037    2552 generic.go:356] "Generic (PLEG): container finished" podID="49c6c8fb-505f-4207-817c-8171b84d8f58" containerID="b5c4b656f031376efdf3300fd81c92415a2cf3c54f0fb0f9dad00b17522c1f5c" exitCode=0
Sep 16 19:18:26.647520 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:26.647110    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d" event={"ID":"49c6c8fb-505f-4207-817c-8171b84d8f58","Type":"ContainerDied","Data":"b5c4b656f031376efdf3300fd81c92415a2cf3c54f0fb0f9dad00b17522c1f5c"}
Sep 16 19:18:27.653141 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:27.653108    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d" event={"ID":"49c6c8fb-505f-4207-817c-8171b84d8f58","Type":"ContainerStarted","Data":"01f231f22a5939a1c7b24489b372fd5c1091dac42110b7068649c71eb477c147"}
Sep 16 19:18:27.653508 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:27.653166    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d" event={"ID":"49c6c8fb-505f-4207-817c-8171b84d8f58","Type":"ContainerStarted","Data":"76b278387c9edc7e85ee704f4b2e37c4508a754cd9505a9e3ae8a45acca4df9d"}
Sep 16 19:18:27.653508 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:27.653345    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:27.674533 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:27.674482    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d" podStartSLOduration=3.822265627 podStartE2EDuration="7.674464207s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.85219858s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:27.673246364 +0000 UTC m=+2303.567433633" watchObservedRunningTime="2026-09-16 19:18:27.674464207 +0000 UTC m=+2303.568651477"
Sep 16 19:18:28.662817 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:28.662784    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:34.528853 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:34.528814    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:18:34.672573 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:34.672543    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:18:36.620253 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:36.620217    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:18:36.620636 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:18:36.620419    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:18:44.528799 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:44.528756    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:18:49.619832 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:49.619792    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:18:49.620255 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:18:49.619968    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:18:54.528677 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:18:54.528633    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:19:02.620059 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:02.620014    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:19:02.620579 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:19:02.620241    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:19:04.528656 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:04.528620    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:19:04.676749 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:04.676716    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:19:10.640598 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:10.640567    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"]
Sep 16 19:19:10.641019 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:10.640775    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d" podUID="49c6c8fb-505f-4207-817c-8171b84d8f58" containerName="kserve-container" containerID="cri-o://76b278387c9edc7e85ee704f4b2e37c4508a754cd9505a9e3ae8a45acca4df9d" gracePeriod=30
Sep 16 19:19:10.641019 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:10.640842    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d" podUID="49c6c8fb-505f-4207-817c-8171b84d8f58" containerName="kube-rbac-proxy" containerID="cri-o://01f231f22a5939a1c7b24489b372fd5c1091dac42110b7068649c71eb477c147" gracePeriod=30
Sep 16 19:19:10.859377 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:10.859348    2552 generic.go:356] "Generic (PLEG): container finished" podID="49c6c8fb-505f-4207-817c-8171b84d8f58" containerID="01f231f22a5939a1c7b24489b372fd5c1091dac42110b7068649c71eb477c147" exitCode=2
Sep 16 19:19:10.859500 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:10.859421    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d" event={"ID":"49c6c8fb-505f-4207-817c-8171b84d8f58","Type":"ContainerDied","Data":"01f231f22a5939a1c7b24489b372fd5c1091dac42110b7068649c71eb477c147"}
Sep 16 19:19:14.528855 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:14.528816    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:19:14.667647 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:14.667611    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d" podUID="49c6c8fb-505f-4207-817c-8171b84d8f58" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 16 19:19:14.673611 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:14.673584    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d" podUID="49c6c8fb-505f-4207-817c-8171b84d8f58" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.51:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 19:19:15.880553 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:15.880525    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"]
Sep 16 19:19:15.883953 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:15.883937    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:15.886435 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:15.886412    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 16 19:19:15.886540 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:15.886417    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 19:19:15.897867 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:15.897845    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"]
Sep 16 19:19:16.069546 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.069519    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3344add-63ce-467f-a3a2-1cc47f4949c9-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.069696 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.069573    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3344add-63ce-467f-a3a2-1cc47f4949c9-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.069696 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.069593    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3344add-63ce-467f-a3a2-1cc47f4949c9-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.069696 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.069664    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3344add-63ce-467f-a3a2-1cc47f4949c9-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.069820 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.069700    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8zd\" (UniqueName: \"kubernetes.io/projected/b3344add-63ce-467f-a3a2-1cc47f4949c9-kube-api-access-8w8zd\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.170295 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.170269    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3344add-63ce-467f-a3a2-1cc47f4949c9-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.170385 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.170304    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3344add-63ce-467f-a3a2-1cc47f4949c9-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.170385 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.170348    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3344add-63ce-467f-a3a2-1cc47f4949c9-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.170465 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.170383    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8zd\" (UniqueName: \"kubernetes.io/projected/b3344add-63ce-467f-a3a2-1cc47f4949c9-kube-api-access-8w8zd\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.170465 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.170432    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3344add-63ce-467f-a3a2-1cc47f4949c9-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.170753 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.170669    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3344add-63ce-467f-a3a2-1cc47f4949c9-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.171032 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.171006    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3344add-63ce-467f-a3a2-1cc47f4949c9-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.171170 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.171057    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3344add-63ce-467f-a3a2-1cc47f4949c9-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.172838 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.172811    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3344add-63ce-467f-a3a2-1cc47f4949c9-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.181526 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.181500    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8zd\" (UniqueName: \"kubernetes.io/projected/b3344add-63ce-467f-a3a2-1cc47f4949c9-kube-api-access-8w8zd\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.194295 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.194272    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:16.287750 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.287727    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:19:16.473149 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.473067    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49c6c8fb-505f-4207-817c-8171b84d8f58-cabundle-cert\") pod \"49c6c8fb-505f-4207-817c-8171b84d8f58\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") "
Sep 16 19:19:16.473149 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.473120    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96524\" (UniqueName: \"kubernetes.io/projected/49c6c8fb-505f-4207-817c-8171b84d8f58-kube-api-access-96524\") pod \"49c6c8fb-505f-4207-817c-8171b84d8f58\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") "
Sep 16 19:19:16.473149 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.473144    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c6c8fb-505f-4207-817c-8171b84d8f58-proxy-tls\") pod \"49c6c8fb-505f-4207-817c-8171b84d8f58\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") "
Sep 16 19:19:16.473425 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.473197    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49c6c8fb-505f-4207-817c-8171b84d8f58-kube-rbac-proxy-sar-config\") pod \"49c6c8fb-505f-4207-817c-8171b84d8f58\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") "
Sep 16 19:19:16.473425 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.473224    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49c6c8fb-505f-4207-817c-8171b84d8f58-kserve-provision-location\") pod \"49c6c8fb-505f-4207-817c-8171b84d8f58\" (UID: \"49c6c8fb-505f-4207-817c-8171b84d8f58\") "
Sep 16 19:19:16.473541 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.473517    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c6c8fb-505f-4207-817c-8171b84d8f58-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "49c6c8fb-505f-4207-817c-8171b84d8f58" (UID: "49c6c8fb-505f-4207-817c-8171b84d8f58"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:16.473598 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.473559    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c6c8fb-505f-4207-817c-8171b84d8f58-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "49c6c8fb-505f-4207-817c-8171b84d8f58" (UID: "49c6c8fb-505f-4207-817c-8171b84d8f58"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:16.473665 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.473641    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c6c8fb-505f-4207-817c-8171b84d8f58-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "49c6c8fb-505f-4207-817c-8171b84d8f58" (UID: "49c6c8fb-505f-4207-817c-8171b84d8f58"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:16.475246 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.475227    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c6c8fb-505f-4207-817c-8171b84d8f58-kube-api-access-96524" (OuterVolumeSpecName: "kube-api-access-96524") pod "49c6c8fb-505f-4207-817c-8171b84d8f58" (UID: "49c6c8fb-505f-4207-817c-8171b84d8f58"). InnerVolumeSpecName "kube-api-access-96524". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:16.475306 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.475267    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c6c8fb-505f-4207-817c-8171b84d8f58-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "49c6c8fb-505f-4207-817c-8171b84d8f58" (UID: "49c6c8fb-505f-4207-817c-8171b84d8f58"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:16.483895 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.483869    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"]
Sep 16 19:19:16.486534 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:19:16.486514    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3344add_63ce_467f_a3a2_1cc47f4949c9.slice/crio-a1c80e91997a4b5711d2b624cfc5f02da1a8186c00633325ba2693c061d5eb2e WatchSource:0}: Error finding container a1c80e91997a4b5711d2b624cfc5f02da1a8186c00633325ba2693c061d5eb2e: Status 404 returned error can't find the container with id a1c80e91997a4b5711d2b624cfc5f02da1a8186c00633325ba2693c061d5eb2e
Sep 16 19:19:16.574736 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.574703    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49c6c8fb-505f-4207-817c-8171b84d8f58-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:19:16.574736 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.574729    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49c6c8fb-505f-4207-817c-8171b84d8f58-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:19:16.574875 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.574743    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49c6c8fb-505f-4207-817c-8171b84d8f58-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:19:16.574875 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.574756    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96524\" (UniqueName: \"kubernetes.io/projected/49c6c8fb-505f-4207-817c-8171b84d8f58-kube-api-access-96524\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:19:16.574875 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.574768    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c6c8fb-505f-4207-817c-8171b84d8f58-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:19:16.620218 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.620186    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:19:16.620396 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:19:16.620374    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:19:16.888924 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.888884    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv" event={"ID":"b3344add-63ce-467f-a3a2-1cc47f4949c9","Type":"ContainerStarted","Data":"d0f5fa5b512ec571ef9025fb28a4944ab35d4be950a342be77661a61b5043b67"}
Sep 16 19:19:16.889435 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.888958    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv" event={"ID":"b3344add-63ce-467f-a3a2-1cc47f4949c9","Type":"ContainerStarted","Data":"a1c80e91997a4b5711d2b624cfc5f02da1a8186c00633325ba2693c061d5eb2e"}
Sep 16 19:19:16.891054 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.891028    2552 generic.go:356] "Generic (PLEG): container finished" podID="49c6c8fb-505f-4207-817c-8171b84d8f58" containerID="76b278387c9edc7e85ee704f4b2e37c4508a754cd9505a9e3ae8a45acca4df9d" exitCode=0
Sep 16 19:19:16.891192 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.891063    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d" event={"ID":"49c6c8fb-505f-4207-817c-8171b84d8f58","Type":"ContainerDied","Data":"76b278387c9edc7e85ee704f4b2e37c4508a754cd9505a9e3ae8a45acca4df9d"}
Sep 16 19:19:16.891192 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.891075    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"
Sep 16 19:19:16.891192 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.891127    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d" event={"ID":"49c6c8fb-505f-4207-817c-8171b84d8f58","Type":"ContainerDied","Data":"cb7d2e6ebae4dbf5e9c70cdf602905f291e3a7a6c5db9da6d72fb967498c376d"}
Sep 16 19:19:16.891192 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.891156    2552 scope.go:117] "RemoveContainer" containerID="01f231f22a5939a1c7b24489b372fd5c1091dac42110b7068649c71eb477c147"
Sep 16 19:19:16.901318 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.901301    2552 scope.go:117] "RemoveContainer" containerID="76b278387c9edc7e85ee704f4b2e37c4508a754cd9505a9e3ae8a45acca4df9d"
Sep 16 19:19:16.908924 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.908868    2552 scope.go:117] "RemoveContainer" containerID="b5c4b656f031376efdf3300fd81c92415a2cf3c54f0fb0f9dad00b17522c1f5c"
Sep 16 19:19:16.916320 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.916296    2552 scope.go:117] "RemoveContainer" containerID="01f231f22a5939a1c7b24489b372fd5c1091dac42110b7068649c71eb477c147"
Sep 16 19:19:16.916601 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:19:16.916579    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f231f22a5939a1c7b24489b372fd5c1091dac42110b7068649c71eb477c147\": container with ID starting with 01f231f22a5939a1c7b24489b372fd5c1091dac42110b7068649c71eb477c147 not found: ID does not exist" containerID="01f231f22a5939a1c7b24489b372fd5c1091dac42110b7068649c71eb477c147"
Sep 16 19:19:16.916668 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.916613    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f231f22a5939a1c7b24489b372fd5c1091dac42110b7068649c71eb477c147"} err="failed to get container status \"01f231f22a5939a1c7b24489b372fd5c1091dac42110b7068649c71eb477c147\": rpc error: code = NotFound desc = could not find container \"01f231f22a5939a1c7b24489b372fd5c1091dac42110b7068649c71eb477c147\": container with ID starting with 01f231f22a5939a1c7b24489b372fd5c1091dac42110b7068649c71eb477c147 not found: ID does not exist"
Sep 16 19:19:16.916668 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.916651    2552 scope.go:117] "RemoveContainer" containerID="76b278387c9edc7e85ee704f4b2e37c4508a754cd9505a9e3ae8a45acca4df9d"
Sep 16 19:19:16.916921 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:19:16.916903    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b278387c9edc7e85ee704f4b2e37c4508a754cd9505a9e3ae8a45acca4df9d\": container with ID starting with 76b278387c9edc7e85ee704f4b2e37c4508a754cd9505a9e3ae8a45acca4df9d not found: ID does not exist" containerID="76b278387c9edc7e85ee704f4b2e37c4508a754cd9505a9e3ae8a45acca4df9d"
Sep 16 19:19:16.916971 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.916928    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b278387c9edc7e85ee704f4b2e37c4508a754cd9505a9e3ae8a45acca4df9d"} err="failed to get container status \"76b278387c9edc7e85ee704f4b2e37c4508a754cd9505a9e3ae8a45acca4df9d\": rpc error: code = NotFound desc = could not find container \"76b278387c9edc7e85ee704f4b2e37c4508a754cd9505a9e3ae8a45acca4df9d\": container with ID starting with 76b278387c9edc7e85ee704f4b2e37c4508a754cd9505a9e3ae8a45acca4df9d not found: ID does not exist"
Sep 16 19:19:16.916971 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.916944    2552 scope.go:117] "RemoveContainer" containerID="b5c4b656f031376efdf3300fd81c92415a2cf3c54f0fb0f9dad00b17522c1f5c"
Sep 16 19:19:16.917239 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:19:16.917212    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c4b656f031376efdf3300fd81c92415a2cf3c54f0fb0f9dad00b17522c1f5c\": container with ID starting with b5c4b656f031376efdf3300fd81c92415a2cf3c54f0fb0f9dad00b17522c1f5c not found: ID does not exist" containerID="b5c4b656f031376efdf3300fd81c92415a2cf3c54f0fb0f9dad00b17522c1f5c"
Sep 16 19:19:16.917339 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.917241    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c4b656f031376efdf3300fd81c92415a2cf3c54f0fb0f9dad00b17522c1f5c"} err="failed to get container status \"b5c4b656f031376efdf3300fd81c92415a2cf3c54f0fb0f9dad00b17522c1f5c\": rpc error: code = NotFound desc = could not find container \"b5c4b656f031376efdf3300fd81c92415a2cf3c54f0fb0f9dad00b17522c1f5c\": container with ID starting with b5c4b656f031376efdf3300fd81c92415a2cf3c54f0fb0f9dad00b17522c1f5c not found: ID does not exist"
Sep 16 19:19:16.930344 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.930319    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"]
Sep 16 19:19:16.935025 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:16.935004    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d"]
Sep 16 19:19:18.625592 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:18.625560    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49c6c8fb-505f-4207-817c-8171b84d8f58" path="/var/lib/kubelet/pods/49c6c8fb-505f-4207-817c-8171b84d8f58/volumes"
Sep 16 19:19:20.911339 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:20.911303    2552 generic.go:356] "Generic (PLEG): container finished" podID="b3344add-63ce-467f-a3a2-1cc47f4949c9" containerID="d0f5fa5b512ec571ef9025fb28a4944ab35d4be950a342be77661a61b5043b67" exitCode=0
Sep 16 19:19:20.911704 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:20.911377    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv" event={"ID":"b3344add-63ce-467f-a3a2-1cc47f4949c9","Type":"ContainerDied","Data":"d0f5fa5b512ec571ef9025fb28a4944ab35d4be950a342be77661a61b5043b67"}
Sep 16 19:19:21.770345 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:21.770309    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"]
Sep 16 19:19:21.770597 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:21.770550    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kserve-container" containerID="cri-o://491cad9d7e4c4aaf80c88dc5866e1a60ce61eaa50083ab2a580c0ccb94f1b11e" gracePeriod=30
Sep 16 19:19:21.770654 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:21.770589    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kube-rbac-proxy" containerID="cri-o://fda739ce5d3e4ee1be9fe314e5d75652d3ea3aa496ac82e4594e7af7ece68398" gracePeriod=30
Sep 16 19:19:21.918599 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:21.918575    2552 generic.go:356] "Generic (PLEG): container finished" podID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerID="fda739ce5d3e4ee1be9fe314e5d75652d3ea3aa496ac82e4594e7af7ece68398" exitCode=2
Sep 16 19:19:21.918903 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:21.918654    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" event={"ID":"d8627bb9-53de-4aad-9a02-ca265ff5a3c3","Type":"ContainerDied","Data":"fda739ce5d3e4ee1be9fe314e5d75652d3ea3aa496ac82e4594e7af7ece68398"}
Sep 16 19:19:21.920826 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:21.920804    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv" event={"ID":"b3344add-63ce-467f-a3a2-1cc47f4949c9","Type":"ContainerStarted","Data":"f601bcb9c1886e7203d0aff3c86926e4b150a575c16834058fc0e45460a7a587"}
Sep 16 19:19:21.920935 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:21.920862    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv" event={"ID":"b3344add-63ce-467f-a3a2-1cc47f4949c9","Type":"ContainerStarted","Data":"f96dd7f4515069b0dd791cb694d538689700b8ddfe0d14537a859c201305f8b4"}
Sep 16 19:19:21.921119 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:21.921083    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:21.945464 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:21.945421    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv" podStartSLOduration=3.215187343 podStartE2EDuration="6.945408252s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.730220909s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:21.94339757 +0000 UTC m=+2357.837584835" watchObservedRunningTime="2026-09-16 19:19:21.945408252 +0000 UTC m=+2357.839595521"
Sep 16 19:19:22.925874 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:22.925843    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:24.522877 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:24.522835    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 16 19:19:24.528178 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:24.528153    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:19:26.217986 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.217952    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:19:26.244528 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.244502    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kube-rbac-proxy-sar-config\") pod \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") "
Sep 16 19:19:26.244658 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.244564    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kserve-provision-location\") pod \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") "
Sep 16 19:19:26.244658 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.244603    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-proxy-tls\") pod \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") "
Sep 16 19:19:26.244775 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.244665    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8lj6\" (UniqueName: \"kubernetes.io/projected/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kube-api-access-p8lj6\") pod \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") "
Sep 16 19:19:26.244775 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.244707    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-cabundle-cert\") pod \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\" (UID: \"d8627bb9-53de-4aad-9a02-ca265ff5a3c3\") "
Sep 16 19:19:26.244879 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.244825    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d8627bb9-53de-4aad-9a02-ca265ff5a3c3" (UID: "d8627bb9-53de-4aad-9a02-ca265ff5a3c3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:26.244879 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.244835    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d8627bb9-53de-4aad-9a02-ca265ff5a3c3" (UID: "d8627bb9-53de-4aad-9a02-ca265ff5a3c3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:26.245003 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.244975    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:19:26.245064 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.245001    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:19:26.245152 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.245121    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d8627bb9-53de-4aad-9a02-ca265ff5a3c3" (UID: "d8627bb9-53de-4aad-9a02-ca265ff5a3c3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:26.246710 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.246687    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d8627bb9-53de-4aad-9a02-ca265ff5a3c3" (UID: "d8627bb9-53de-4aad-9a02-ca265ff5a3c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:26.246802 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.246738    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kube-api-access-p8lj6" (OuterVolumeSpecName: "kube-api-access-p8lj6") pod "d8627bb9-53de-4aad-9a02-ca265ff5a3c3" (UID: "d8627bb9-53de-4aad-9a02-ca265ff5a3c3"). InnerVolumeSpecName "kube-api-access-p8lj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:26.345973 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.345913    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:19:26.345973 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.345937    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8lj6\" (UniqueName: \"kubernetes.io/projected/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-kube-api-access-p8lj6\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:19:26.345973 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.345946    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8627bb9-53de-4aad-9a02-ca265ff5a3c3-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:19:26.946321 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.946230    2552 generic.go:356] "Generic (PLEG): container finished" podID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerID="491cad9d7e4c4aaf80c88dc5866e1a60ce61eaa50083ab2a580c0ccb94f1b11e" exitCode=0
Sep 16 19:19:26.946321 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.946288    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"
Sep 16 19:19:26.946321 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.946312    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" event={"ID":"d8627bb9-53de-4aad-9a02-ca265ff5a3c3","Type":"ContainerDied","Data":"491cad9d7e4c4aaf80c88dc5866e1a60ce61eaa50083ab2a580c0ccb94f1b11e"}
Sep 16 19:19:26.946551 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.946375    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs" event={"ID":"d8627bb9-53de-4aad-9a02-ca265ff5a3c3","Type":"ContainerDied","Data":"62e4107eed16fdd7015140926064552f966a22f5a18a73280d3048e0ff2174a5"}
Sep 16 19:19:26.946551 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.946402    2552 scope.go:117] "RemoveContainer" containerID="fda739ce5d3e4ee1be9fe314e5d75652d3ea3aa496ac82e4594e7af7ece68398"
Sep 16 19:19:26.954566 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.954547    2552 scope.go:117] "RemoveContainer" containerID="491cad9d7e4c4aaf80c88dc5866e1a60ce61eaa50083ab2a580c0ccb94f1b11e"
Sep 16 19:19:26.961815 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.961799    2552 scope.go:117] "RemoveContainer" containerID="bbf481a9b1156ccf00b122596dbdc702b3e3cf2e8d2fd0f22caed1ab34d2223f"
Sep 16 19:19:26.968892 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.968876    2552 scope.go:117] "RemoveContainer" containerID="fda739ce5d3e4ee1be9fe314e5d75652d3ea3aa496ac82e4594e7af7ece68398"
Sep 16 19:19:26.969162 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:19:26.969144    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda739ce5d3e4ee1be9fe314e5d75652d3ea3aa496ac82e4594e7af7ece68398\": container with ID starting with fda739ce5d3e4ee1be9fe314e5d75652d3ea3aa496ac82e4594e7af7ece68398 not found: ID does not exist" containerID="fda739ce5d3e4ee1be9fe314e5d75652d3ea3aa496ac82e4594e7af7ece68398"
Sep 16 19:19:26.969228 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.969169    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda739ce5d3e4ee1be9fe314e5d75652d3ea3aa496ac82e4594e7af7ece68398"} err="failed to get container status \"fda739ce5d3e4ee1be9fe314e5d75652d3ea3aa496ac82e4594e7af7ece68398\": rpc error: code = NotFound desc = could not find container \"fda739ce5d3e4ee1be9fe314e5d75652d3ea3aa496ac82e4594e7af7ece68398\": container with ID starting with fda739ce5d3e4ee1be9fe314e5d75652d3ea3aa496ac82e4594e7af7ece68398 not found: ID does not exist"
Sep 16 19:19:26.969228 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.969183    2552 scope.go:117] "RemoveContainer" containerID="491cad9d7e4c4aaf80c88dc5866e1a60ce61eaa50083ab2a580c0ccb94f1b11e"
Sep 16 19:19:26.969416 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:19:26.969401    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491cad9d7e4c4aaf80c88dc5866e1a60ce61eaa50083ab2a580c0ccb94f1b11e\": container with ID starting with 491cad9d7e4c4aaf80c88dc5866e1a60ce61eaa50083ab2a580c0ccb94f1b11e not found: ID does not exist" containerID="491cad9d7e4c4aaf80c88dc5866e1a60ce61eaa50083ab2a580c0ccb94f1b11e"
Sep 16 19:19:26.969457 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.969420    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491cad9d7e4c4aaf80c88dc5866e1a60ce61eaa50083ab2a580c0ccb94f1b11e"} err="failed to get container status \"491cad9d7e4c4aaf80c88dc5866e1a60ce61eaa50083ab2a580c0ccb94f1b11e\": rpc error: code = NotFound desc = could not find container \"491cad9d7e4c4aaf80c88dc5866e1a60ce61eaa50083ab2a580c0ccb94f1b11e\": container with ID starting with 491cad9d7e4c4aaf80c88dc5866e1a60ce61eaa50083ab2a580c0ccb94f1b11e not found: ID does not exist"
Sep 16 19:19:26.969457 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.969434    2552 scope.go:117] "RemoveContainer" containerID="bbf481a9b1156ccf00b122596dbdc702b3e3cf2e8d2fd0f22caed1ab34d2223f"
Sep 16 19:19:26.969641 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:19:26.969626    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf481a9b1156ccf00b122596dbdc702b3e3cf2e8d2fd0f22caed1ab34d2223f\": container with ID starting with bbf481a9b1156ccf00b122596dbdc702b3e3cf2e8d2fd0f22caed1ab34d2223f not found: ID does not exist" containerID="bbf481a9b1156ccf00b122596dbdc702b3e3cf2e8d2fd0f22caed1ab34d2223f"
Sep 16 19:19:26.969682 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.969645    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf481a9b1156ccf00b122596dbdc702b3e3cf2e8d2fd0f22caed1ab34d2223f"} err="failed to get container status \"bbf481a9b1156ccf00b122596dbdc702b3e3cf2e8d2fd0f22caed1ab34d2223f\": rpc error: code = NotFound desc = could not find container \"bbf481a9b1156ccf00b122596dbdc702b3e3cf2e8d2fd0f22caed1ab34d2223f\": container with ID starting with bbf481a9b1156ccf00b122596dbdc702b3e3cf2e8d2fd0f22caed1ab34d2223f not found: ID does not exist"
Sep 16 19:19:26.972740 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.972719    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"]
Sep 16 19:19:26.982836 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:26.982816    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs"]
Sep 16 19:19:28.625303 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:28.625272    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" path="/var/lib/kubelet/pods/d8627bb9-53de-4aad-9a02-ca265ff5a3c3/volumes"
Sep 16 19:19:28.934909 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:28.934829    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:30.619697 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:30.619669    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:19:30.620078 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:19:30.619839    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:19:45.619604 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:45.619570    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:19:45.619984 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:19:45.619745    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:19:58.939791 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:58.939759    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:19:59.620208 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:19:59.620180    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:19:59.620385 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:19:59.620348    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:20:05.887791 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:05.887756    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"]
Sep 16 19:20:05.888218 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:05.887989    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv" podUID="b3344add-63ce-467f-a3a2-1cc47f4949c9" containerName="kserve-container" containerID="cri-o://f96dd7f4515069b0dd791cb694d538689700b8ddfe0d14537a859c201305f8b4" gracePeriod=30
Sep 16 19:20:05.888218 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:05.888061    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv" podUID="b3344add-63ce-467f-a3a2-1cc47f4949c9" containerName="kube-rbac-proxy" containerID="cri-o://f601bcb9c1886e7203d0aff3c86926e4b150a575c16834058fc0e45460a7a587" gracePeriod=30
Sep 16 19:20:06.124022 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:06.123993    2552 generic.go:356] "Generic (PLEG): container finished" podID="b3344add-63ce-467f-a3a2-1cc47f4949c9" containerID="f601bcb9c1886e7203d0aff3c86926e4b150a575c16834058fc0e45460a7a587" exitCode=2
Sep 16 19:20:06.124159 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:06.124051    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv" event={"ID":"b3344add-63ce-467f-a3a2-1cc47f4949c9","Type":"ContainerDied","Data":"f601bcb9c1886e7203d0aff3c86926e4b150a575c16834058fc0e45460a7a587"}
Sep 16 19:20:08.931166 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:08.931126    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv" podUID="b3344add-63ce-467f-a3a2-1cc47f4949c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Sep 16 19:20:08.935403 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:08.935381    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv" podUID="b3344add-63ce-467f-a3a2-1cc47f4949c9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.52:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 16 19:20:11.620254 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.620227    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:20:11.620535 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:20:11.620448    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:20:11.631056 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.631033    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:20:11.704833 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.704778    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3344add-63ce-467f-a3a2-1cc47f4949c9-kube-rbac-proxy-sar-config\") pod \"b3344add-63ce-467f-a3a2-1cc47f4949c9\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") "
Sep 16 19:20:11.704833 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.704811    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3344add-63ce-467f-a3a2-1cc47f4949c9-cabundle-cert\") pod \"b3344add-63ce-467f-a3a2-1cc47f4949c9\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") "
Sep 16 19:20:11.704964 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.704853    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3344add-63ce-467f-a3a2-1cc47f4949c9-proxy-tls\") pod \"b3344add-63ce-467f-a3a2-1cc47f4949c9\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") "
Sep 16 19:20:11.704964 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.704891    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3344add-63ce-467f-a3a2-1cc47f4949c9-kserve-provision-location\") pod \"b3344add-63ce-467f-a3a2-1cc47f4949c9\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") "
Sep 16 19:20:11.704964 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.704951    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w8zd\" (UniqueName: \"kubernetes.io/projected/b3344add-63ce-467f-a3a2-1cc47f4949c9-kube-api-access-8w8zd\") pod \"b3344add-63ce-467f-a3a2-1cc47f4949c9\" (UID: \"b3344add-63ce-467f-a3a2-1cc47f4949c9\") "
Sep 16 19:20:11.705217 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.705196    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3344add-63ce-467f-a3a2-1cc47f4949c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b3344add-63ce-467f-a3a2-1cc47f4949c9" (UID: "b3344add-63ce-467f-a3a2-1cc47f4949c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:11.705296 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.705213    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3344add-63ce-467f-a3a2-1cc47f4949c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3344add-63ce-467f-a3a2-1cc47f4949c9" (UID: "b3344add-63ce-467f-a3a2-1cc47f4949c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:11.705362 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.705289    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3344add-63ce-467f-a3a2-1cc47f4949c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b3344add-63ce-467f-a3a2-1cc47f4949c9" (UID: "b3344add-63ce-467f-a3a2-1cc47f4949c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:11.705427 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.705395    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3344add-63ce-467f-a3a2-1cc47f4949c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:20:11.705427 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.705413    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3344add-63ce-467f-a3a2-1cc47f4949c9-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:20:11.706968 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.706937    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3344add-63ce-467f-a3a2-1cc47f4949c9-kube-api-access-8w8zd" (OuterVolumeSpecName: "kube-api-access-8w8zd") pod "b3344add-63ce-467f-a3a2-1cc47f4949c9" (UID: "b3344add-63ce-467f-a3a2-1cc47f4949c9"). InnerVolumeSpecName "kube-api-access-8w8zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:11.707059 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.706986    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3344add-63ce-467f-a3a2-1cc47f4949c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b3344add-63ce-467f-a3a2-1cc47f4949c9" (UID: "b3344add-63ce-467f-a3a2-1cc47f4949c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:11.806004 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.805980    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3344add-63ce-467f-a3a2-1cc47f4949c9-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:20:11.806004 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.806001    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3344add-63ce-467f-a3a2-1cc47f4949c9-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:20:11.806152 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:11.806010    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w8zd\" (UniqueName: \"kubernetes.io/projected/b3344add-63ce-467f-a3a2-1cc47f4949c9-kube-api-access-8w8zd\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:20:12.153918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.153886    2552 generic.go:356] "Generic (PLEG): container finished" podID="b3344add-63ce-467f-a3a2-1cc47f4949c9" containerID="f96dd7f4515069b0dd791cb694d538689700b8ddfe0d14537a859c201305f8b4" exitCode=0
Sep 16 19:20:12.154079 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.153950    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"
Sep 16 19:20:12.154079 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.153962    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv" event={"ID":"b3344add-63ce-467f-a3a2-1cc47f4949c9","Type":"ContainerDied","Data":"f96dd7f4515069b0dd791cb694d538689700b8ddfe0d14537a859c201305f8b4"}
Sep 16 19:20:12.154079 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.154031    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv" event={"ID":"b3344add-63ce-467f-a3a2-1cc47f4949c9","Type":"ContainerDied","Data":"a1c80e91997a4b5711d2b624cfc5f02da1a8186c00633325ba2693c061d5eb2e"}
Sep 16 19:20:12.154079 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.154059    2552 scope.go:117] "RemoveContainer" containerID="f601bcb9c1886e7203d0aff3c86926e4b150a575c16834058fc0e45460a7a587"
Sep 16 19:20:12.162918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.162899    2552 scope.go:117] "RemoveContainer" containerID="f96dd7f4515069b0dd791cb694d538689700b8ddfe0d14537a859c201305f8b4"
Sep 16 19:20:12.170329 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.170310    2552 scope.go:117] "RemoveContainer" containerID="d0f5fa5b512ec571ef9025fb28a4944ab35d4be950a342be77661a61b5043b67"
Sep 16 19:20:12.177173 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.177155    2552 scope.go:117] "RemoveContainer" containerID="f601bcb9c1886e7203d0aff3c86926e4b150a575c16834058fc0e45460a7a587"
Sep 16 19:20:12.177430 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:20:12.177411    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f601bcb9c1886e7203d0aff3c86926e4b150a575c16834058fc0e45460a7a587\": container with ID starting with f601bcb9c1886e7203d0aff3c86926e4b150a575c16834058fc0e45460a7a587 not found: ID does not exist" containerID="f601bcb9c1886e7203d0aff3c86926e4b150a575c16834058fc0e45460a7a587"
Sep 16 19:20:12.177485 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.177437    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f601bcb9c1886e7203d0aff3c86926e4b150a575c16834058fc0e45460a7a587"} err="failed to get container status \"f601bcb9c1886e7203d0aff3c86926e4b150a575c16834058fc0e45460a7a587\": rpc error: code = NotFound desc = could not find container \"f601bcb9c1886e7203d0aff3c86926e4b150a575c16834058fc0e45460a7a587\": container with ID starting with f601bcb9c1886e7203d0aff3c86926e4b150a575c16834058fc0e45460a7a587 not found: ID does not exist"
Sep 16 19:20:12.177485 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.177453    2552 scope.go:117] "RemoveContainer" containerID="f96dd7f4515069b0dd791cb694d538689700b8ddfe0d14537a859c201305f8b4"
Sep 16 19:20:12.177714 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:20:12.177690    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f96dd7f4515069b0dd791cb694d538689700b8ddfe0d14537a859c201305f8b4\": container with ID starting with f96dd7f4515069b0dd791cb694d538689700b8ddfe0d14537a859c201305f8b4 not found: ID does not exist" containerID="f96dd7f4515069b0dd791cb694d538689700b8ddfe0d14537a859c201305f8b4"
Sep 16 19:20:12.177763 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.177712    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96dd7f4515069b0dd791cb694d538689700b8ddfe0d14537a859c201305f8b4"} err="failed to get container status \"f96dd7f4515069b0dd791cb694d538689700b8ddfe0d14537a859c201305f8b4\": rpc error: code = NotFound desc = could not find container \"f96dd7f4515069b0dd791cb694d538689700b8ddfe0d14537a859c201305f8b4\": container with ID starting with f96dd7f4515069b0dd791cb694d538689700b8ddfe0d14537a859c201305f8b4 not found: ID does not exist"
Sep 16 19:20:12.177763 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.177725    2552 scope.go:117] "RemoveContainer" containerID="d0f5fa5b512ec571ef9025fb28a4944ab35d4be950a342be77661a61b5043b67"
Sep 16 19:20:12.177998 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:20:12.177967    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f5fa5b512ec571ef9025fb28a4944ab35d4be950a342be77661a61b5043b67\": container with ID starting with d0f5fa5b512ec571ef9025fb28a4944ab35d4be950a342be77661a61b5043b67 not found: ID does not exist" containerID="d0f5fa5b512ec571ef9025fb28a4944ab35d4be950a342be77661a61b5043b67"
Sep 16 19:20:12.178055 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.177995    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f5fa5b512ec571ef9025fb28a4944ab35d4be950a342be77661a61b5043b67"} err="failed to get container status \"d0f5fa5b512ec571ef9025fb28a4944ab35d4be950a342be77661a61b5043b67\": rpc error: code = NotFound desc = could not find container \"d0f5fa5b512ec571ef9025fb28a4944ab35d4be950a342be77661a61b5043b67\": container with ID starting with d0f5fa5b512ec571ef9025fb28a4944ab35d4be950a342be77661a61b5043b67 not found: ID does not exist"
Sep 16 19:20:12.182143 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.182110    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"]
Sep 16 19:20:12.189810 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.189792    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv"]
Sep 16 19:20:12.624804 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:12.624775    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3344add-63ce-467f-a3a2-1cc47f4949c9" path="/var/lib/kubelet/pods/b3344add-63ce-467f-a3a2-1cc47f4949c9/volumes"
Sep 16 19:20:23.619847 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:23.619806    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:20:23.620302 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:20:23.619979    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:20:35.620303 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:35.620225    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:20:35.620736 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:20:35.620400    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:20:50.620217 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:20:50.620186    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:20:50.620622 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:20:50.620353    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:21:02.620333 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:02.620301    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:21:02.623040 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:02.620584    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:21:03.376550 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:03.376519    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerStarted","Data":"fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"}
Sep 16 19:21:03.376754 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:03.376735    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:21:07.482164 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482126    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"]
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482463    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kube-rbac-proxy"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482475    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kube-rbac-proxy"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482482    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49c6c8fb-505f-4207-817c-8171b84d8f58" containerName="kube-rbac-proxy"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482488    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="49c6c8fb-505f-4207-817c-8171b84d8f58" containerName="kube-rbac-proxy"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482496    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3344add-63ce-467f-a3a2-1cc47f4949c9" containerName="storage-initializer"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482502    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3344add-63ce-467f-a3a2-1cc47f4949c9" containerName="storage-initializer"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482514    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="storage-initializer"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482520    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="storage-initializer"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482527    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3344add-63ce-467f-a3a2-1cc47f4949c9" containerName="kube-rbac-proxy"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482534    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3344add-63ce-467f-a3a2-1cc47f4949c9" containerName="kube-rbac-proxy"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482541    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49c6c8fb-505f-4207-817c-8171b84d8f58" containerName="storage-initializer"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482546    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="49c6c8fb-505f-4207-817c-8171b84d8f58" containerName="storage-initializer"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482552    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49c6c8fb-505f-4207-817c-8171b84d8f58" containerName="kserve-container"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482557    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="49c6c8fb-505f-4207-817c-8171b84d8f58" containerName="kserve-container"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482567    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3344add-63ce-467f-a3a2-1cc47f4949c9" containerName="kserve-container"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482572    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3344add-63ce-467f-a3a2-1cc47f4949c9" containerName="kserve-container"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482580    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kserve-container"
Sep 16 19:21:07.482612 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482585    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kserve-container"
Sep 16 19:21:07.483183 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482655    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kube-rbac-proxy"
Sep 16 19:21:07.483183 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482670    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8627bb9-53de-4aad-9a02-ca265ff5a3c3" containerName="kserve-container"
Sep 16 19:21:07.483183 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482678    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="49c6c8fb-505f-4207-817c-8171b84d8f58" containerName="kube-rbac-proxy"
Sep 16 19:21:07.483183 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482688    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3344add-63ce-467f-a3a2-1cc47f4949c9" containerName="kube-rbac-proxy"
Sep 16 19:21:07.483183 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482697    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="49c6c8fb-505f-4207-817c-8171b84d8f58" containerName="kserve-container"
Sep 16 19:21:07.483183 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.482703    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3344add-63ce-467f-a3a2-1cc47f4949c9" containerName="kserve-container"
Sep 16 19:21:07.485551 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.485534    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:07.488338 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.488318    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:21:07.488445 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.488368    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:21:07.488445 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.488368    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:21:07.488774 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.488751    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 16 19:21:07.489196 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.489182    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:21:07.489255 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.489226    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:21:07.489713 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.489699    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:21:07.501700 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.501672    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"]
Sep 16 19:21:07.612789 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.612762    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b51a08e-d891-4388-8c9b-cc3626cabea9-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:07.612916 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.612795    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b51a08e-d891-4388-8c9b-cc3626cabea9-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:07.612916 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.612830    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b51a08e-d891-4388-8c9b-cc3626cabea9-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:07.612916 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.612858    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b51a08e-d891-4388-8c9b-cc3626cabea9-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:07.612916 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.612910    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24w9n\" (UniqueName: \"kubernetes.io/projected/4b51a08e-d891-4388-8c9b-cc3626cabea9-kube-api-access-24w9n\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:07.713967 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.713942    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24w9n\" (UniqueName: \"kubernetes.io/projected/4b51a08e-d891-4388-8c9b-cc3626cabea9-kube-api-access-24w9n\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:07.714117 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.714009    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b51a08e-d891-4388-8c9b-cc3626cabea9-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:07.714117 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.714036    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b51a08e-d891-4388-8c9b-cc3626cabea9-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:07.714252 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.714121    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b51a08e-d891-4388-8c9b-cc3626cabea9-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:07.714252 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.714143    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b51a08e-d891-4388-8c9b-cc3626cabea9-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:07.714252 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:21:07.714241    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 16 19:21:07.714412 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:21:07.714296    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b51a08e-d891-4388-8c9b-cc3626cabea9-proxy-tls podName:4b51a08e-d891-4388-8c9b-cc3626cabea9 nodeName:}" failed. No retries permitted until 2026-09-16 19:21:08.214277052 +0000 UTC m=+2464.108464313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4b51a08e-d891-4388-8c9b-cc3626cabea9-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" (UID: "4b51a08e-d891-4388-8c9b-cc3626cabea9") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 16 19:21:07.714481 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.714435    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b51a08e-d891-4388-8c9b-cc3626cabea9-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:07.714818 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.714797    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b51a08e-d891-4388-8c9b-cc3626cabea9-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:07.714882 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.714835    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b51a08e-d891-4388-8c9b-cc3626cabea9-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:07.726803 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:07.726778    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24w9n\" (UniqueName: \"kubernetes.io/projected/4b51a08e-d891-4388-8c9b-cc3626cabea9-kube-api-access-24w9n\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:08.217913 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:08.217874    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b51a08e-d891-4388-8c9b-cc3626cabea9-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:08.220409 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:08.220388    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b51a08e-d891-4388-8c9b-cc3626cabea9-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:08.395451 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:08.395415    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:08.680734 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:08.680704    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"]
Sep 16 19:21:08.683725 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:21:08.683698    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b51a08e_d891_4388_8c9b_cc3626cabea9.slice/crio-c88fd1080e38634cfdc6823a26a8c4f4ff643a6c0e85c45ba6f0d5ecc1a53a61 WatchSource:0}: Error finding container c88fd1080e38634cfdc6823a26a8c4f4ff643a6c0e85c45ba6f0d5ecc1a53a61: Status 404 returned error can't find the container with id c88fd1080e38634cfdc6823a26a8c4f4ff643a6c0e85c45ba6f0d5ecc1a53a61
Sep 16 19:21:09.404140 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:09.404104    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" event={"ID":"4b51a08e-d891-4388-8c9b-cc3626cabea9","Type":"ContainerStarted","Data":"71b62377a4003765a2c88e575ea91d6fa0b322ee4e93fdbe58a0779195fe2442"}
Sep 16 19:21:09.404326 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:09.404166    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" event={"ID":"4b51a08e-d891-4388-8c9b-cc3626cabea9","Type":"ContainerStarted","Data":"c88fd1080e38634cfdc6823a26a8c4f4ff643a6c0e85c45ba6f0d5ecc1a53a61"}
Sep 16 19:21:13.422967 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:13.422929    2552 generic.go:356] "Generic (PLEG): container finished" podID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerID="71b62377a4003765a2c88e575ea91d6fa0b322ee4e93fdbe58a0779195fe2442" exitCode=0
Sep 16 19:21:13.423405 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:13.423000    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" event={"ID":"4b51a08e-d891-4388-8c9b-cc3626cabea9","Type":"ContainerDied","Data":"71b62377a4003765a2c88e575ea91d6fa0b322ee4e93fdbe58a0779195fe2442"}
Sep 16 19:21:20.459037 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:20.459003    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" event={"ID":"4b51a08e-d891-4388-8c9b-cc3626cabea9","Type":"ContainerStarted","Data":"a371763656c0bb950a210f642691a699f65c997c8bfd88ebe4d166b5894012ae"}
Sep 16 19:21:20.459414 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:20.459065    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" event={"ID":"4b51a08e-d891-4388-8c9b-cc3626cabea9","Type":"ContainerStarted","Data":"d9efa6dac91718f4f682fd2f4eb3f6a7a96b21052b16313a74465d45822b5945"}
Sep 16 19:21:20.459414 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:20.459282    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:20.480697 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:20.480654    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" podStartSLOduration=3.321691783 podStartE2EDuration="13.480641833s" totalImagesPullingTime="6.439071157s" totalInitContainerRuntime="3.719878893s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:20.479748177 +0000 UTC m=+2476.373935446" watchObservedRunningTime="2026-09-16 19:21:20.480641833 +0000 UTC m=+2476.374829102"
Sep 16 19:21:21.468892 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:21.468855    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:21.470639 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:21.470613    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 16 19:21:22.472871 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:22.472827    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 16 19:21:27.478473 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:27.478439    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:21:27.479102 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:27.479053    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 16 19:21:34.383405 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:34.383373    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:21:36.387839 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.387803    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"]
Sep 16 19:21:36.391452 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.391434    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:36.394809 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.394788    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 16 19:21:36.395471 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.395453    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:21:36.403952 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.403930    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"]
Sep 16 19:21:36.447693 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.447662    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:36.447780 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.447706    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxqt8\" (UniqueName: \"kubernetes.io/projected/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kube-api-access-xxqt8\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:36.447780 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.447734    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:36.447859 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.447806    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:36.447859 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.447840    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:36.548844 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.548819    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxqt8\" (UniqueName: \"kubernetes.io/projected/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kube-api-access-xxqt8\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:36.548968 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.548860    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:36.548968 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.548887    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:36.548968 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.548927    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:36.549132 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.549007    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:36.549186 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:21:36.549150    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 16 19:21:36.549230 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:21:36.549216    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-proxy-tls podName:bc7aafba-a0b3-45c6-a7c9-79d1526c3d51 nodeName:}" failed. No retries permitted until 2026-09-16 19:21:37.04919375 +0000 UTC m=+2492.943381012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs" (UID: "bc7aafba-a0b3-45c6-a7c9-79d1526c3d51") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 16 19:21:36.549285 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.549240    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:36.549672 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.549656    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:36.549710 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.549659    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:36.560169 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:36.560139    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxqt8\" (UniqueName: \"kubernetes.io/projected/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kube-api-access-xxqt8\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:37.052784 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:37.052741    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:37.055289 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:37.055252    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:37.301122 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:37.301075    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:37.479315 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:37.479278    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 16 19:21:37.584824 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:37.584747    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"]
Sep 16 19:21:37.587826 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:21:37.587801    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7aafba_a0b3_45c6_a7c9_79d1526c3d51.slice/crio-7df983f74fdf64abf74660b98bf59449100cbfc733eadc5e31ef9c96c287d502 WatchSource:0}: Error finding container 7df983f74fdf64abf74660b98bf59449100cbfc733eadc5e31ef9c96c287d502: Status 404 returned error can't find the container with id 7df983f74fdf64abf74660b98bf59449100cbfc733eadc5e31ef9c96c287d502
Sep 16 19:21:38.541252 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:38.541216    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs" event={"ID":"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51","Type":"ContainerStarted","Data":"11d26d153e8880564724805dad00b7786aa4fe4d206374f69aceacdd3c66f474"}
Sep 16 19:21:38.541677 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:38.541287    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs" event={"ID":"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51","Type":"ContainerStarted","Data":"7df983f74fdf64abf74660b98bf59449100cbfc733eadc5e31ef9c96c287d502"}
Sep 16 19:21:41.556419 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:41.556390    2552 generic.go:356] "Generic (PLEG): container finished" podID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerID="11d26d153e8880564724805dad00b7786aa4fe4d206374f69aceacdd3c66f474" exitCode=0
Sep 16 19:21:41.556718 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:41.556463    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs" event={"ID":"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51","Type":"ContainerDied","Data":"11d26d153e8880564724805dad00b7786aa4fe4d206374f69aceacdd3c66f474"}
Sep 16 19:21:42.563221 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:42.563187    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs" event={"ID":"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51","Type":"ContainerStarted","Data":"236c6cafbaf11885ab65d2be5ed274f9209f5b92b3ca49b2f5070c4b9e06ba43"}
Sep 16 19:21:42.563593 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:42.563279    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs" event={"ID":"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51","Type":"ContainerStarted","Data":"26c08d66aa6def4ff93be6f838b31a180a90e9b5d90eb86dcc4924470c068916"}
Sep 16 19:21:42.563593 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:42.563472    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:42.586702 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:42.586641    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs" podStartSLOduration=2.816338212 podStartE2EDuration="6.586624666s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.770286454s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:42.586205259 +0000 UTC m=+2498.480392528" watchObservedRunningTime="2026-09-16 19:21:42.586624666 +0000 UTC m=+2498.480811958"
Sep 16 19:21:43.567245 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:43.567212    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:47.479567 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:47.479527    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 16 19:21:49.577321 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:49.577147    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:21:57.479902 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:21:57.479859    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 16 19:22:07.479336 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:07.479291    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 16 19:22:17.479295 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:17.479254    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 16 19:22:19.632849 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:19.632801    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs" podUID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:22:27.479398 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:27.479356    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 16 19:22:29.579873 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:29.579843    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:22:36.467736 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:36.467703    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"]
Sep 16 19:22:36.468164 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:36.467934    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs" podUID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerName="kserve-container" containerID="cri-o://26c08d66aa6def4ff93be6f838b31a180a90e9b5d90eb86dcc4924470c068916" gracePeriod=30
Sep 16 19:22:36.468164 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:36.467979    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs" podUID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerName="kube-rbac-proxy" containerID="cri-o://236c6cafbaf11885ab65d2be5ed274f9209f5b92b3ca49b2f5070c4b9e06ba43" gracePeriod=30
Sep 16 19:22:36.801385 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:36.801300    2552 generic.go:356] "Generic (PLEG): container finished" podID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerID="236c6cafbaf11885ab65d2be5ed274f9209f5b92b3ca49b2f5070c4b9e06ba43" exitCode=2
Sep 16 19:22:36.801385 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:36.801366    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs" event={"ID":"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51","Type":"ContainerDied","Data":"236c6cafbaf11885ab65d2be5ed274f9209f5b92b3ca49b2f5070c4b9e06ba43"}
Sep 16 19:22:37.480238 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:37.480206    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:22:39.572665 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:39.572627    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs" podUID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Sep 16 19:22:39.578193 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:39.578162    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs" podUID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.54:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 19:22:41.619351 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.619317    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"]
Sep 16 19:22:41.621978 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.621957    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:41.624296 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.624271    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 16 19:22:41.624400 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.624311    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:22:41.634213 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.634190    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"]
Sep 16 19:22:41.754876 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.754840    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:41.755004 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.754918    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vcn7\" (UniqueName: \"kubernetes.io/projected/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kube-api-access-8vcn7\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:41.755004 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.754961    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:41.755004 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.754991    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:41.755141 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.755010    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:41.855651 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.855622    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vcn7\" (UniqueName: \"kubernetes.io/projected/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kube-api-access-8vcn7\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:41.855793 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.855667    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:41.855793 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.855696    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:41.855793 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.855721    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:41.855914 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:22:41.855804    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 16 19:22:41.855914 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.855833    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:41.855914 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:22:41.855863    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-proxy-tls podName:4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88 nodeName:}" failed. No retries permitted until 2026-09-16 19:22:42.355845438 +0000 UTC m=+2558.250032714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-bd4kg" (UID: "4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 16 19:22:41.856139 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.856116    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:41.856558 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.856538    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:41.856641 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.856626    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:41.865203 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:41.865183    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vcn7\" (UniqueName: \"kubernetes.io/projected/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kube-api-access-8vcn7\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:42.360707 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:42.360672    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:42.363138 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:42.363105    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bd4kg\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:42.531507 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:42.531473    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:22:42.805920 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:42.805892    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"]
Sep 16 19:22:42.809182 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:22:42.809153    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0af7ef_7edc_4d3a_b525_71a5a8cf9d88.slice/crio-e8152896df4fd78347ba977ee7f3bbdc84d5f494fd5c4d95ddfa6b600a7eae87 WatchSource:0}: Error finding container e8152896df4fd78347ba977ee7f3bbdc84d5f494fd5c4d95ddfa6b600a7eae87: Status 404 returned error can't find the container with id e8152896df4fd78347ba977ee7f3bbdc84d5f494fd5c4d95ddfa6b600a7eae87
Sep 16 19:22:42.828148 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:42.828110    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" event={"ID":"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88","Type":"ContainerStarted","Data":"e8152896df4fd78347ba977ee7f3bbdc84d5f494fd5c4d95ddfa6b600a7eae87"}
Sep 16 19:22:42.830421 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:42.830396    2552 generic.go:356] "Generic (PLEG): container finished" podID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerID="26c08d66aa6def4ff93be6f838b31a180a90e9b5d90eb86dcc4924470c068916" exitCode=0
Sep 16 19:22:42.830526 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:42.830438    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs" event={"ID":"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51","Type":"ContainerDied","Data":"26c08d66aa6def4ff93be6f838b31a180a90e9b5d90eb86dcc4924470c068916"}
Sep 16 19:22:42.914859 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:42.914838    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:22:43.066346 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.066266    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-proxy-tls\") pod \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") "
Sep 16 19:22:43.066346 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.066314    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kserve-provision-location\") pod \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") "
Sep 16 19:22:43.066586 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.066349    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxqt8\" (UniqueName: \"kubernetes.io/projected/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kube-api-access-xxqt8\") pod \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") "
Sep 16 19:22:43.066586 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.066379    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-cabundle-cert\") pod \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") "
Sep 16 19:22:43.066586 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.066406    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kube-rbac-proxy-sar-config\") pod \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\" (UID: \"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51\") "
Sep 16 19:22:43.066743 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.066714    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" (UID: "bc7aafba-a0b3-45c6-a7c9-79d1526c3d51"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:43.066812 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.066785    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" (UID: "bc7aafba-a0b3-45c6-a7c9-79d1526c3d51"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:43.066812 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.066800    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" (UID: "bc7aafba-a0b3-45c6-a7c9-79d1526c3d51"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:43.068973 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.068942    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" (UID: "bc7aafba-a0b3-45c6-a7c9-79d1526c3d51"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:43.069174 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.068972    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kube-api-access-xxqt8" (OuterVolumeSpecName: "kube-api-access-xxqt8") pod "bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" (UID: "bc7aafba-a0b3-45c6-a7c9-79d1526c3d51"). InnerVolumeSpecName "kube-api-access-xxqt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:43.167588 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.167558    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.167588 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.167583    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.167588 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.167593    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxqt8\" (UniqueName: \"kubernetes.io/projected/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kube-api-access-xxqt8\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.167783 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.167602    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.167783 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.167610    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.836670 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.836626    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" event={"ID":"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88","Type":"ContainerStarted","Data":"e299b5dd7e10632332b17e719de3578f071bcce688952753e99645aa9d15e029"}
Sep 16 19:22:43.838605 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.838577    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs" event={"ID":"bc7aafba-a0b3-45c6-a7c9-79d1526c3d51","Type":"ContainerDied","Data":"7df983f74fdf64abf74660b98bf59449100cbfc733eadc5e31ef9c96c287d502"}
Sep 16 19:22:43.838727 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.838633    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"
Sep 16 19:22:43.838727 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.838638    2552 scope.go:117] "RemoveContainer" containerID="236c6cafbaf11885ab65d2be5ed274f9209f5b92b3ca49b2f5070c4b9e06ba43"
Sep 16 19:22:43.849333 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.849312    2552 scope.go:117] "RemoveContainer" containerID="26c08d66aa6def4ff93be6f838b31a180a90e9b5d90eb86dcc4924470c068916"
Sep 16 19:22:43.856759 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.856741    2552 scope.go:117] "RemoveContainer" containerID="11d26d153e8880564724805dad00b7786aa4fe4d206374f69aceacdd3c66f474"
Sep 16 19:22:43.875102 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.875065    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"]
Sep 16 19:22:43.880999 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:43.880978    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs"]
Sep 16 19:22:44.625819 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:44.625789    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" path="/var/lib/kubelet/pods/bc7aafba-a0b3-45c6-a7c9-79d1526c3d51/volumes"
Sep 16 19:22:46.855479 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:46.855445    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerID="e299b5dd7e10632332b17e719de3578f071bcce688952753e99645aa9d15e029" exitCode=0
Sep 16 19:22:46.855899 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:46.855487    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" event={"ID":"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88","Type":"ContainerDied","Data":"e299b5dd7e10632332b17e719de3578f071bcce688952753e99645aa9d15e029"}
Sep 16 19:22:48.692070 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:48.692036    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"]
Sep 16 19:22:48.692566 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:48.692400    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kserve-container" containerID="cri-o://d9efa6dac91718f4f682fd2f4eb3f6a7a96b21052b16313a74465d45822b5945" gracePeriod=30
Sep 16 19:22:48.692566 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:48.692445    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kube-rbac-proxy" containerID="cri-o://a371763656c0bb950a210f642691a699f65c997c8bfd88ebe4d166b5894012ae" gracePeriod=30
Sep 16 19:22:48.871195 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:48.871162    2552 generic.go:356] "Generic (PLEG): container finished" podID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerID="a371763656c0bb950a210f642691a699f65c997c8bfd88ebe4d166b5894012ae" exitCode=2
Sep 16 19:22:48.871337 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:48.871215    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" event={"ID":"4b51a08e-d891-4388-8c9b-cc3626cabea9","Type":"ContainerDied","Data":"a371763656c0bb950a210f642691a699f65c997c8bfd88ebe4d166b5894012ae"}
Sep 16 19:22:52.473512 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:52.473468    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 16 19:22:53.755668 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.755643    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:22:53.857253 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.857166    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b51a08e-d891-4388-8c9b-cc3626cabea9-cabundle-cert\") pod \"4b51a08e-d891-4388-8c9b-cc3626cabea9\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") "
Sep 16 19:22:53.857253 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.857218    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b51a08e-d891-4388-8c9b-cc3626cabea9-proxy-tls\") pod \"4b51a08e-d891-4388-8c9b-cc3626cabea9\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") "
Sep 16 19:22:53.857483 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.857273    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24w9n\" (UniqueName: \"kubernetes.io/projected/4b51a08e-d891-4388-8c9b-cc3626cabea9-kube-api-access-24w9n\") pod \"4b51a08e-d891-4388-8c9b-cc3626cabea9\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") "
Sep 16 19:22:53.857483 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.857314    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b51a08e-d891-4388-8c9b-cc3626cabea9-kserve-provision-location\") pod \"4b51a08e-d891-4388-8c9b-cc3626cabea9\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") "
Sep 16 19:22:53.857483 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.857389    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b51a08e-d891-4388-8c9b-cc3626cabea9-kube-rbac-proxy-sar-config\") pod \"4b51a08e-d891-4388-8c9b-cc3626cabea9\" (UID: \"4b51a08e-d891-4388-8c9b-cc3626cabea9\") "
Sep 16 19:22:53.857641 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.857512    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b51a08e-d891-4388-8c9b-cc3626cabea9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b51a08e-d891-4388-8c9b-cc3626cabea9" (UID: "4b51a08e-d891-4388-8c9b-cc3626cabea9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:53.857641 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.857625    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b51a08e-d891-4388-8c9b-cc3626cabea9-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:22:53.857835 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.857788    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b51a08e-d891-4388-8c9b-cc3626cabea9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b51a08e-d891-4388-8c9b-cc3626cabea9" (UID: "4b51a08e-d891-4388-8c9b-cc3626cabea9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:53.857978 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.857946    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b51a08e-d891-4388-8c9b-cc3626cabea9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b51a08e-d891-4388-8c9b-cc3626cabea9" (UID: "4b51a08e-d891-4388-8c9b-cc3626cabea9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:53.860083 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.860028    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b51a08e-d891-4388-8c9b-cc3626cabea9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b51a08e-d891-4388-8c9b-cc3626cabea9" (UID: "4b51a08e-d891-4388-8c9b-cc3626cabea9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:53.860222 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.860181    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b51a08e-d891-4388-8c9b-cc3626cabea9-kube-api-access-24w9n" (OuterVolumeSpecName: "kube-api-access-24w9n") pod "4b51a08e-d891-4388-8c9b-cc3626cabea9" (UID: "4b51a08e-d891-4388-8c9b-cc3626cabea9"). InnerVolumeSpecName "kube-api-access-24w9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:53.897920 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.897883    2552 generic.go:356] "Generic (PLEG): container finished" podID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerID="d9efa6dac91718f4f682fd2f4eb3f6a7a96b21052b16313a74465d45822b5945" exitCode=0
Sep 16 19:22:53.898062 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.898032    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" event={"ID":"4b51a08e-d891-4388-8c9b-cc3626cabea9","Type":"ContainerDied","Data":"d9efa6dac91718f4f682fd2f4eb3f6a7a96b21052b16313a74465d45822b5945"}
Sep 16 19:22:53.898161 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.898118    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k" event={"ID":"4b51a08e-d891-4388-8c9b-cc3626cabea9","Type":"ContainerDied","Data":"c88fd1080e38634cfdc6823a26a8c4f4ff643a6c0e85c45ba6f0d5ecc1a53a61"}
Sep 16 19:22:53.898218 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.898159    2552 scope.go:117] "RemoveContainer" containerID="a371763656c0bb950a210f642691a699f65c997c8bfd88ebe4d166b5894012ae"
Sep 16 19:22:53.898587 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.898342    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"
Sep 16 19:22:53.912985 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.912965    2552 scope.go:117] "RemoveContainer" containerID="d9efa6dac91718f4f682fd2f4eb3f6a7a96b21052b16313a74465d45822b5945"
Sep 16 19:22:53.921300 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.921278    2552 scope.go:117] "RemoveContainer" containerID="71b62377a4003765a2c88e575ea91d6fa0b322ee4e93fdbe58a0779195fe2442"
Sep 16 19:22:53.930066 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.930033    2552 scope.go:117] "RemoveContainer" containerID="a371763656c0bb950a210f642691a699f65c997c8bfd88ebe4d166b5894012ae"
Sep 16 19:22:53.930406 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:22:53.930378    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a371763656c0bb950a210f642691a699f65c997c8bfd88ebe4d166b5894012ae\": container with ID starting with a371763656c0bb950a210f642691a699f65c997c8bfd88ebe4d166b5894012ae not found: ID does not exist" containerID="a371763656c0bb950a210f642691a699f65c997c8bfd88ebe4d166b5894012ae"
Sep 16 19:22:53.930481 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.930416    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a371763656c0bb950a210f642691a699f65c997c8bfd88ebe4d166b5894012ae"} err="failed to get container status \"a371763656c0bb950a210f642691a699f65c997c8bfd88ebe4d166b5894012ae\": rpc error: code = NotFound desc = could not find container \"a371763656c0bb950a210f642691a699f65c997c8bfd88ebe4d166b5894012ae\": container with ID starting with a371763656c0bb950a210f642691a699f65c997c8bfd88ebe4d166b5894012ae not found: ID does not exist"
Sep 16 19:22:53.930481 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.930442    2552 scope.go:117] "RemoveContainer" containerID="d9efa6dac91718f4f682fd2f4eb3f6a7a96b21052b16313a74465d45822b5945"
Sep 16 19:22:53.930753 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:22:53.930707    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9efa6dac91718f4f682fd2f4eb3f6a7a96b21052b16313a74465d45822b5945\": container with ID starting with d9efa6dac91718f4f682fd2f4eb3f6a7a96b21052b16313a74465d45822b5945 not found: ID does not exist" containerID="d9efa6dac91718f4f682fd2f4eb3f6a7a96b21052b16313a74465d45822b5945"
Sep 16 19:22:53.930753 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.930745    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9efa6dac91718f4f682fd2f4eb3f6a7a96b21052b16313a74465d45822b5945"} err="failed to get container status \"d9efa6dac91718f4f682fd2f4eb3f6a7a96b21052b16313a74465d45822b5945\": rpc error: code = NotFound desc = could not find container \"d9efa6dac91718f4f682fd2f4eb3f6a7a96b21052b16313a74465d45822b5945\": container with ID starting with d9efa6dac91718f4f682fd2f4eb3f6a7a96b21052b16313a74465d45822b5945 not found: ID does not exist"
Sep 16 19:22:53.930905 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.930762    2552 scope.go:117] "RemoveContainer" containerID="71b62377a4003765a2c88e575ea91d6fa0b322ee4e93fdbe58a0779195fe2442"
Sep 16 19:22:53.931138 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:22:53.931113    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b62377a4003765a2c88e575ea91d6fa0b322ee4e93fdbe58a0779195fe2442\": container with ID starting with 71b62377a4003765a2c88e575ea91d6fa0b322ee4e93fdbe58a0779195fe2442 not found: ID does not exist" containerID="71b62377a4003765a2c88e575ea91d6fa0b322ee4e93fdbe58a0779195fe2442"
Sep 16 19:22:53.931235 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.931147    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b62377a4003765a2c88e575ea91d6fa0b322ee4e93fdbe58a0779195fe2442"} err="failed to get container status \"71b62377a4003765a2c88e575ea91d6fa0b322ee4e93fdbe58a0779195fe2442\": rpc error: code = NotFound desc = could not find container \"71b62377a4003765a2c88e575ea91d6fa0b322ee4e93fdbe58a0779195fe2442\": container with ID starting with 71b62377a4003765a2c88e575ea91d6fa0b322ee4e93fdbe58a0779195fe2442 not found: ID does not exist"
Sep 16 19:22:53.944675 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.944651    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"]
Sep 16 19:22:53.947030 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947009    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"]
Sep 16 19:22:53.947483 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947468    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kube-rbac-proxy"
Sep 16 19:22:53.947483 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947486    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kube-rbac-proxy"
Sep 16 19:22:53.947587 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947497    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerName="kube-rbac-proxy"
Sep 16 19:22:53.947587 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947506    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerName="kube-rbac-proxy"
Sep 16 19:22:53.947587 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947518    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="storage-initializer"
Sep 16 19:22:53.947587 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947527    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="storage-initializer"
Sep 16 19:22:53.947587 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947548    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerName="storage-initializer"
Sep 16 19:22:53.947587 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947556    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerName="storage-initializer"
Sep 16 19:22:53.947587 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947581    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kserve-container"
Sep 16 19:22:53.947838 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947589    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kserve-container"
Sep 16 19:22:53.947838 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947600    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerName="kserve-container"
Sep 16 19:22:53.947838 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947608    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerName="kserve-container"
Sep 16 19:22:53.947838 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947683    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kserve-container"
Sep 16 19:22:53.947838 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947695    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" containerName="kube-rbac-proxy"
Sep 16 19:22:53.947838 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947704    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerName="kserve-container"
Sep 16 19:22:53.947838 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.947715    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc7aafba-a0b3-45c6-a7c9-79d1526c3d51" containerName="kube-rbac-proxy"
Sep 16 19:22:53.952572 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.952553    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k"]
Sep 16 19:22:53.952665 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.952654    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:53.954963 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.954946    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 16 19:22:53.955072 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.955054    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:22:53.959050 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.959028    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b51a08e-d891-4388-8c9b-cc3626cabea9-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:22:53.959149 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.959055    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b51a08e-d891-4388-8c9b-cc3626cabea9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:22:53.959149 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.959065    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b51a08e-d891-4388-8c9b-cc3626cabea9-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:22:53.959149 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.959076    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24w9n\" (UniqueName: \"kubernetes.io/projected/4b51a08e-d891-4388-8c9b-cc3626cabea9-kube-api-access-24w9n\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:22:53.970624 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:53.970592    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"]
Sep 16 19:22:54.059805 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.059768    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkf96\" (UniqueName: \"kubernetes.io/projected/b5e377d8-8054-4656-8a14-71d32fa1bec2-kube-api-access-rkf96\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.059972 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.059832    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5e377d8-8054-4656-8a14-71d32fa1bec2-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.059972 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.059880    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5e377d8-8054-4656-8a14-71d32fa1bec2-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.059972 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.059931    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5e377d8-8054-4656-8a14-71d32fa1bec2-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.059972 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.059964    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5e377d8-8054-4656-8a14-71d32fa1bec2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.160744 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.160713    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5e377d8-8054-4656-8a14-71d32fa1bec2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.160911 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.160773    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkf96\" (UniqueName: \"kubernetes.io/projected/b5e377d8-8054-4656-8a14-71d32fa1bec2-kube-api-access-rkf96\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.160911 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.160828    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5e377d8-8054-4656-8a14-71d32fa1bec2-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.160911 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.160855    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5e377d8-8054-4656-8a14-71d32fa1bec2-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.161122 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.160909    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5e377d8-8054-4656-8a14-71d32fa1bec2-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.161122 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:22:54.161039    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 16 19:22:54.161263 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:22:54.161155    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e377d8-8054-4656-8a14-71d32fa1bec2-proxy-tls podName:b5e377d8-8054-4656-8a14-71d32fa1bec2 nodeName:}" failed. No retries permitted until 2026-09-16 19:22:54.66113228 +0000 UTC m=+2570.555319545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b5e377d8-8054-4656-8a14-71d32fa1bec2-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6" (UID: "b5e377d8-8054-4656-8a14-71d32fa1bec2") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 16 19:22:54.161263 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.161173    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5e377d8-8054-4656-8a14-71d32fa1bec2-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.161472 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.161439    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5e377d8-8054-4656-8a14-71d32fa1bec2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.161733 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.161714    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5e377d8-8054-4656-8a14-71d32fa1bec2-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.178949 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.178922    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkf96\" (UniqueName: \"kubernetes.io/projected/b5e377d8-8054-4656-8a14-71d32fa1bec2-kube-api-access-rkf96\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.627135 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.627103    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b51a08e-d891-4388-8c9b-cc3626cabea9" path="/var/lib/kubelet/pods/4b51a08e-d891-4388-8c9b-cc3626cabea9/volumes"
Sep 16 19:22:54.664933 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.664905    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5e377d8-8054-4656-8a14-71d32fa1bec2-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.667642 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.667616    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5e377d8-8054-4656-8a14-71d32fa1bec2-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:54.866216 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:54.866163    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:22:55.152281 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:55.152251    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"]
Sep 16 19:22:55.155499 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:22:55.155469    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e377d8_8054_4656_8a14_71d32fa1bec2.slice/crio-dd62712513063fa5c4f2671fb68d21e56a40b9877cb8a50fb412391a462f31c6 WatchSource:0}: Error finding container dd62712513063fa5c4f2671fb68d21e56a40b9877cb8a50fb412391a462f31c6: Status 404 returned error can't find the container with id dd62712513063fa5c4f2671fb68d21e56a40b9877cb8a50fb412391a462f31c6
Sep 16 19:22:55.912282 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:55.912238    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6" event={"ID":"b5e377d8-8054-4656-8a14-71d32fa1bec2","Type":"ContainerStarted","Data":"74e9cc20cb0679c1584bb7d6accd4a0d121bdca11abfa93c06f68dc9088c2ac6"}
Sep 16 19:22:55.912723 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:22:55.912312    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6" event={"ID":"b5e377d8-8054-4656-8a14-71d32fa1bec2","Type":"ContainerStarted","Data":"dd62712513063fa5c4f2671fb68d21e56a40b9877cb8a50fb412391a462f31c6"}
Sep 16 19:23:03.955811 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:03.955772    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" event={"ID":"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88","Type":"ContainerStarted","Data":"3dfb6b3b8852c7db098b3e9da6e75a1ee644d5df9f1eebd76dd78dd2e694e51a"}
Sep 16 19:23:03.956331 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:03.955896    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" event={"ID":"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88","Type":"ContainerStarted","Data":"96f2519804a14ac781903bca3d54392eb1a70d3d75a7bf8a06d8b6dbcdaa5275"}
Sep 16 19:23:03.956331 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:03.956141    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:23:03.957694 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:03.957668    2552 generic.go:356] "Generic (PLEG): container finished" podID="b5e377d8-8054-4656-8a14-71d32fa1bec2" containerID="74e9cc20cb0679c1584bb7d6accd4a0d121bdca11abfa93c06f68dc9088c2ac6" exitCode=0
Sep 16 19:23:03.957810 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:03.957735    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6" event={"ID":"b5e377d8-8054-4656-8a14-71d32fa1bec2","Type":"ContainerDied","Data":"74e9cc20cb0679c1584bb7d6accd4a0d121bdca11abfa93c06f68dc9088c2ac6"}
Sep 16 19:23:03.979957 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:03.979905    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" podStartSLOduration=2.304760675 podStartE2EDuration="22.97988748s" totalImagesPullingTime="16.887030291s" totalInitContainerRuntime="3.788096514s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:03.977509034 +0000 UTC m=+2579.871696304" watchObservedRunningTime="2026-09-16 19:23:03.97988748 +0000 UTC m=+2579.874074749"
Sep 16 19:23:04.966367 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:04.966327    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6" event={"ID":"b5e377d8-8054-4656-8a14-71d32fa1bec2","Type":"ContainerStarted","Data":"49b9c2ebe163e61567ea2e5f614602a28072a6f4ffb78bb561b02f42df88df0e"}
Sep 16 19:23:04.966865 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:04.966389    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6" event={"ID":"b5e377d8-8054-4656-8a14-71d32fa1bec2","Type":"ContainerStarted","Data":"0359894a3585483958237501f6332c2542a7d2a68e259557b232029a1246a079"}
Sep 16 19:23:04.966865 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:04.966595    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:23:04.966865 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:04.966798    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:23:04.966865 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:04.966849    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:23:04.967965 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:04.967931    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 16 19:23:04.987552 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:04.987504    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6" podStartSLOduration=7.7456445 podStartE2EDuration="11.987490032s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.241845532s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:04.986747317 +0000 UTC m=+2580.880934621" watchObservedRunningTime="2026-09-16 19:23:04.987490032 +0000 UTC m=+2580.881677300"
Sep 16 19:23:05.971276 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:05.971232    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 16 19:23:10.977145 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:10.977114    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:23:10.977598 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:10.977579    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:23:10.977676 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:10.977618    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 16 19:23:19.382531 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:19.382483    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.27:8081/readyz\": dial tcp 10.132.0.27:8081: connect: connection refused"
Sep 16 19:23:20.038806 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:20.038765    2552 generic.go:356] "Generic (PLEG): container finished" podID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9" exitCode=1
Sep 16 19:23:20.039002 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:20.038833    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerDied","Data":"fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"}
Sep 16 19:23:20.039002 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:20.038890    2552 scope.go:117] "RemoveContainer" containerID="a9c2e3400d43db0c2931e94dfe98172b1357407dd76f1300aabbe220cf478308"
Sep 16 19:23:20.039236 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:20.039222    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:23:20.039458 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:23:20.039437    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:23:20.978128 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:20.978068    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 16 19:23:24.381993 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:24.381953    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:23:24.382438 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:24.382411    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:23:24.382639 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:23:24.382619    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:23:26.788471 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:26.788380    2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:23:26.788841 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:26.788732    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:23:26.788917 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:23:26.788897    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:23:30.977579 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:30.977531    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 16 19:23:37.619948 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:37.619914    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:23:37.620373 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:23:37.620130    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:23:40.977635 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:40.977589    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 16 19:23:40.981531 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:40.981506    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:23:43.926371 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:43.926338    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"]
Sep 16 19:23:43.926849 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:43.926629    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6" podUID="b5e377d8-8054-4656-8a14-71d32fa1bec2" containerName="kserve-container" containerID="cri-o://0359894a3585483958237501f6332c2542a7d2a68e259557b232029a1246a079" gracePeriod=30
Sep 16 19:23:43.926849 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:43.926705    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6" podUID="b5e377d8-8054-4656-8a14-71d32fa1bec2" containerName="kube-rbac-proxy" containerID="cri-o://49b9c2ebe163e61567ea2e5f614602a28072a6f4ffb78bb561b02f42df88df0e" gracePeriod=30
Sep 16 19:23:44.151200 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:44.151167    2552 generic.go:356] "Generic (PLEG): container finished" podID="b5e377d8-8054-4656-8a14-71d32fa1bec2" containerID="49b9c2ebe163e61567ea2e5f614602a28072a6f4ffb78bb561b02f42df88df0e" exitCode=2
Sep 16 19:23:44.151382 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:44.151240    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6" event={"ID":"b5e377d8-8054-4656-8a14-71d32fa1bec2","Type":"ContainerDied","Data":"49b9c2ebe163e61567ea2e5f614602a28072a6f4ffb78bb561b02f42df88df0e"}
Sep 16 19:23:45.090822 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.090797    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:23:45.157905 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.157866    2552 generic.go:356] "Generic (PLEG): container finished" podID="b5e377d8-8054-4656-8a14-71d32fa1bec2" containerID="0359894a3585483958237501f6332c2542a7d2a68e259557b232029a1246a079" exitCode=0
Sep 16 19:23:45.158162 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.157928    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"
Sep 16 19:23:45.158162 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.157943    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6" event={"ID":"b5e377d8-8054-4656-8a14-71d32fa1bec2","Type":"ContainerDied","Data":"0359894a3585483958237501f6332c2542a7d2a68e259557b232029a1246a079"}
Sep 16 19:23:45.158162 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.158005    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6" event={"ID":"b5e377d8-8054-4656-8a14-71d32fa1bec2","Type":"ContainerDied","Data":"dd62712513063fa5c4f2671fb68d21e56a40b9877cb8a50fb412391a462f31c6"}
Sep 16 19:23:45.158162 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.158056    2552 scope.go:117] "RemoveContainer" containerID="49b9c2ebe163e61567ea2e5f614602a28072a6f4ffb78bb561b02f42df88df0e"
Sep 16 19:23:45.166731 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.166710    2552 scope.go:117] "RemoveContainer" containerID="0359894a3585483958237501f6332c2542a7d2a68e259557b232029a1246a079"
Sep 16 19:23:45.175112 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.175077    2552 scope.go:117] "RemoveContainer" containerID="74e9cc20cb0679c1584bb7d6accd4a0d121bdca11abfa93c06f68dc9088c2ac6"
Sep 16 19:23:45.183011 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.182989    2552 scope.go:117] "RemoveContainer" containerID="49b9c2ebe163e61567ea2e5f614602a28072a6f4ffb78bb561b02f42df88df0e"
Sep 16 19:23:45.183373 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:23:45.183347    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b9c2ebe163e61567ea2e5f614602a28072a6f4ffb78bb561b02f42df88df0e\": container with ID starting with 49b9c2ebe163e61567ea2e5f614602a28072a6f4ffb78bb561b02f42df88df0e not found: ID does not exist" containerID="49b9c2ebe163e61567ea2e5f614602a28072a6f4ffb78bb561b02f42df88df0e"
Sep 16 19:23:45.183442 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.183382    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b9c2ebe163e61567ea2e5f614602a28072a6f4ffb78bb561b02f42df88df0e"} err="failed to get container status \"49b9c2ebe163e61567ea2e5f614602a28072a6f4ffb78bb561b02f42df88df0e\": rpc error: code = NotFound desc = could not find container \"49b9c2ebe163e61567ea2e5f614602a28072a6f4ffb78bb561b02f42df88df0e\": container with ID starting with 49b9c2ebe163e61567ea2e5f614602a28072a6f4ffb78bb561b02f42df88df0e not found: ID does not exist"
Sep 16 19:23:45.183442 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.183402    2552 scope.go:117] "RemoveContainer" containerID="0359894a3585483958237501f6332c2542a7d2a68e259557b232029a1246a079"
Sep 16 19:23:45.183661 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:23:45.183645    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0359894a3585483958237501f6332c2542a7d2a68e259557b232029a1246a079\": container with ID starting with 0359894a3585483958237501f6332c2542a7d2a68e259557b232029a1246a079 not found: ID does not exist" containerID="0359894a3585483958237501f6332c2542a7d2a68e259557b232029a1246a079"
Sep 16 19:23:45.183701 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.183667    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0359894a3585483958237501f6332c2542a7d2a68e259557b232029a1246a079"} err="failed to get container status \"0359894a3585483958237501f6332c2542a7d2a68e259557b232029a1246a079\": rpc error: code = NotFound desc = could not find container \"0359894a3585483958237501f6332c2542a7d2a68e259557b232029a1246a079\": container with ID starting with 0359894a3585483958237501f6332c2542a7d2a68e259557b232029a1246a079 not found: ID does not exist"
Sep 16 19:23:45.183701 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.183681    2552 scope.go:117] "RemoveContainer" containerID="74e9cc20cb0679c1584bb7d6accd4a0d121bdca11abfa93c06f68dc9088c2ac6"
Sep 16 19:23:45.183933 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:23:45.183919    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e9cc20cb0679c1584bb7d6accd4a0d121bdca11abfa93c06f68dc9088c2ac6\": container with ID starting with 74e9cc20cb0679c1584bb7d6accd4a0d121bdca11abfa93c06f68dc9088c2ac6 not found: ID does not exist" containerID="74e9cc20cb0679c1584bb7d6accd4a0d121bdca11abfa93c06f68dc9088c2ac6"
Sep 16 19:23:45.183972 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.183937    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e9cc20cb0679c1584bb7d6accd4a0d121bdca11abfa93c06f68dc9088c2ac6"} err="failed to get container status \"74e9cc20cb0679c1584bb7d6accd4a0d121bdca11abfa93c06f68dc9088c2ac6\": rpc error: code = NotFound desc = could not find container \"74e9cc20cb0679c1584bb7d6accd4a0d121bdca11abfa93c06f68dc9088c2ac6\": container with ID starting with 74e9cc20cb0679c1584bb7d6accd4a0d121bdca11abfa93c06f68dc9088c2ac6 not found: ID does not exist"
Sep 16 19:23:45.212926 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.212903    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5e377d8-8054-4656-8a14-71d32fa1bec2-cabundle-cert\") pod \"b5e377d8-8054-4656-8a14-71d32fa1bec2\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") "
Sep 16 19:23:45.212987 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.212937    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5e377d8-8054-4656-8a14-71d32fa1bec2-kserve-provision-location\") pod \"b5e377d8-8054-4656-8a14-71d32fa1bec2\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") "
Sep 16 19:23:45.212987 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.212959    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkf96\" (UniqueName: \"kubernetes.io/projected/b5e377d8-8054-4656-8a14-71d32fa1bec2-kube-api-access-rkf96\") pod \"b5e377d8-8054-4656-8a14-71d32fa1bec2\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") "
Sep 16 19:23:45.213060 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.213002    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5e377d8-8054-4656-8a14-71d32fa1bec2-kube-rbac-proxy-sar-config\") pod \"b5e377d8-8054-4656-8a14-71d32fa1bec2\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") "
Sep 16 19:23:45.213060 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.213037    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5e377d8-8054-4656-8a14-71d32fa1bec2-proxy-tls\") pod \"b5e377d8-8054-4656-8a14-71d32fa1bec2\" (UID: \"b5e377d8-8054-4656-8a14-71d32fa1bec2\") "
Sep 16 19:23:45.213328 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.213301    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5e377d8-8054-4656-8a14-71d32fa1bec2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5e377d8-8054-4656-8a14-71d32fa1bec2" (UID: "b5e377d8-8054-4656-8a14-71d32fa1bec2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:45.213328 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.213319    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e377d8-8054-4656-8a14-71d32fa1bec2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5e377d8-8054-4656-8a14-71d32fa1bec2" (UID: "b5e377d8-8054-4656-8a14-71d32fa1bec2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:23:45.213417 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.213363    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e377d8-8054-4656-8a14-71d32fa1bec2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5e377d8-8054-4656-8a14-71d32fa1bec2" (UID: "b5e377d8-8054-4656-8a14-71d32fa1bec2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:23:45.215127 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.215079    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e377d8-8054-4656-8a14-71d32fa1bec2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5e377d8-8054-4656-8a14-71d32fa1bec2" (UID: "b5e377d8-8054-4656-8a14-71d32fa1bec2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:45.215233 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.215132    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e377d8-8054-4656-8a14-71d32fa1bec2-kube-api-access-rkf96" (OuterVolumeSpecName: "kube-api-access-rkf96") pod "b5e377d8-8054-4656-8a14-71d32fa1bec2" (UID: "b5e377d8-8054-4656-8a14-71d32fa1bec2"). InnerVolumeSpecName "kube-api-access-rkf96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:45.314296 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.314246    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5e377d8-8054-4656-8a14-71d32fa1bec2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:23:45.314296 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.314285    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5e377d8-8054-4656-8a14-71d32fa1bec2-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:23:45.314296 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.314299    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5e377d8-8054-4656-8a14-71d32fa1bec2-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:23:45.314296 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.314308    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5e377d8-8054-4656-8a14-71d32fa1bec2-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:23:45.314643 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.314317    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkf96\" (UniqueName: \"kubernetes.io/projected/b5e377d8-8054-4656-8a14-71d32fa1bec2-kube-api-access-rkf96\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:23:45.482698 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.482669    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"]
Sep 16 19:23:45.487487 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:45.487451    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6"]
Sep 16 19:23:46.625408 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:46.625361    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5e377d8-8054-4656-8a14-71d32fa1bec2" path="/var/lib/kubelet/pods/b5e377d8-8054-4656-8a14-71d32fa1bec2/volumes"
Sep 16 19:23:49.118982 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.118937    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"]
Sep 16 19:23:49.119443 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.119319    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5e377d8-8054-4656-8a14-71d32fa1bec2" containerName="kube-rbac-proxy"
Sep 16 19:23:49.119443 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.119331    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5e377d8-8054-4656-8a14-71d32fa1bec2" containerName="kube-rbac-proxy"
Sep 16 19:23:49.119443 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.119342    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5e377d8-8054-4656-8a14-71d32fa1bec2" containerName="kserve-container"
Sep 16 19:23:49.119443 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.119347    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5e377d8-8054-4656-8a14-71d32fa1bec2" containerName="kserve-container"
Sep 16 19:23:49.119443 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.119364    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5e377d8-8054-4656-8a14-71d32fa1bec2" containerName="storage-initializer"
Sep 16 19:23:49.119443 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.119370    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5e377d8-8054-4656-8a14-71d32fa1bec2" containerName="storage-initializer"
Sep 16 19:23:49.119443 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.119423    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5e377d8-8054-4656-8a14-71d32fa1bec2" containerName="kube-rbac-proxy"
Sep 16 19:23:49.119443 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.119434    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5e377d8-8054-4656-8a14-71d32fa1bec2" containerName="kserve-container"
Sep 16 19:23:49.122984 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.122952    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.125429 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.125387    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 16 19:23:49.125601 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.125393    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 19:23:49.134711 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.134653    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"]
Sep 16 19:23:49.249120 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.249048    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.249303 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.249192    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.249303 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.249248    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.249418 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.249305    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.249418 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.249347    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7z6g\" (UniqueName: \"kubernetes.io/projected/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kube-api-access-n7z6g\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.350217 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.350171    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.350416 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.350235    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.350416 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.350271    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.350416 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.350293    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.350416 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.350320    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7z6g\" (UniqueName: \"kubernetes.io/projected/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kube-api-access-n7z6g\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.350779 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.350757    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.351185 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.351163    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.351246 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.351163    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.352829 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.352795    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.360989 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.360955    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7z6g\" (UniqueName: \"kubernetes.io/projected/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kube-api-access-n7z6g\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.436142 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.436067    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:49.620121 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.620051    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:23:49.620275 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:23:49.620237    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:23:49.719654 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:49.719624    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"]
Sep 16 19:23:49.721762 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:23:49.721737    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b1116b_a7db_461f_b4fd_77c74c3fb2fd.slice/crio-92b6502484a4af3787b784b88c0cb4eea78364ffb5f1338c4d094bd32fc982e3 WatchSource:0}: Error finding container 92b6502484a4af3787b784b88c0cb4eea78364ffb5f1338c4d094bd32fc982e3: Status 404 returned error can't find the container with id 92b6502484a4af3787b784b88c0cb4eea78364ffb5f1338c4d094bd32fc982e3
Sep 16 19:23:50.188586 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:50.188540    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8" event={"ID":"04b1116b-a7db-461f-b4fd-77c74c3fb2fd","Type":"ContainerStarted","Data":"6b763fcda43010bdda9d884258ca76138c8103f56785a428e0df1da60f64d4e6"}
Sep 16 19:23:50.189057 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:50.188619    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8" event={"ID":"04b1116b-a7db-461f-b4fd-77c74c3fb2fd","Type":"ContainerStarted","Data":"92b6502484a4af3787b784b88c0cb4eea78364ffb5f1338c4d094bd32fc982e3"}
Sep 16 19:23:50.977933 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:50.977893    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 16 19:23:54.208503 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:54.208454    2552 generic.go:356] "Generic (PLEG): container finished" podID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerID="6b763fcda43010bdda9d884258ca76138c8103f56785a428e0df1da60f64d4e6" exitCode=0
Sep 16 19:23:54.208954 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:54.208529    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8" event={"ID":"04b1116b-a7db-461f-b4fd-77c74c3fb2fd","Type":"ContainerDied","Data":"6b763fcda43010bdda9d884258ca76138c8103f56785a428e0df1da60f64d4e6"}
Sep 16 19:23:55.216575 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:55.216541    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8" event={"ID":"04b1116b-a7db-461f-b4fd-77c74c3fb2fd","Type":"ContainerStarted","Data":"3004497a128ed7a019967ce276fde339f518d317c2b90070667f39f84e93461c"}
Sep 16 19:23:55.217004 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:55.216604    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8" event={"ID":"04b1116b-a7db-461f-b4fd-77c74c3fb2fd","Type":"ContainerStarted","Data":"8f5b3ea0db8dd9e90e84f3860c29350e9547ff3b824f6a5658af78a03b1e0c35"}
Sep 16 19:23:55.217004 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:55.216876    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:55.217004 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:55.216936    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:23:55.218666 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:55.218637    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8" podUID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 16 19:23:55.241364 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:55.241308    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8" podStartSLOduration=2.216230772 podStartE2EDuration="6.241291102s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.02506033s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:55.240190547 +0000 UTC m=+2631.134377818" watchObservedRunningTime="2026-09-16 19:23:55.241291102 +0000 UTC m=+2631.135478371"
Sep 16 19:23:56.221681 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:23:56.221640    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8" podUID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 16 19:24:00.978167 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:00.978120    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 16 19:24:01.226263 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:01.226221    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:24:01.226950 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:01.226917    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8" podUID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 16 19:24:02.620416 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:02.620384    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:24:02.620796 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:24:02.620561    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:24:10.978266 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:10.978232    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:24:11.227261 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:11.227226    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:24:11.716175 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:11.716142    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"]
Sep 16 19:24:11.716444 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:11.716409    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kserve-container" containerID="cri-o://96f2519804a14ac781903bca3d54392eb1a70d3d75a7bf8a06d8b6dbcdaa5275" gracePeriod=30
Sep 16 19:24:11.716605 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:11.716463    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kube-rbac-proxy" containerID="cri-o://3dfb6b3b8852c7db098b3e9da6e75a1ee644d5df9f1eebd76dd78dd2e694e51a" gracePeriod=30
Sep 16 19:24:12.299511 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:12.299475    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerID="3dfb6b3b8852c7db098b3e9da6e75a1ee644d5df9f1eebd76dd78dd2e694e51a" exitCode=2
Sep 16 19:24:12.299920 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:12.299551    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" event={"ID":"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88","Type":"ContainerDied","Data":"3dfb6b3b8852c7db098b3e9da6e75a1ee644d5df9f1eebd76dd78dd2e694e51a"}
Sep 16 19:24:15.680108 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.680062    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:24:15.781190 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.781071    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-proxy-tls\") pod \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") "
Sep 16 19:24:15.781190 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.781182    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kube-rbac-proxy-sar-config\") pod \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") "
Sep 16 19:24:15.781396 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.781237    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kserve-provision-location\") pod \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") "
Sep 16 19:24:15.781396 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.781304    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-cabundle-cert\") pod \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") "
Sep 16 19:24:15.781396 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.781334    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vcn7\" (UniqueName: \"kubernetes.io/projected/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kube-api-access-8vcn7\") pod \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\" (UID: \"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88\") "
Sep 16 19:24:15.781623 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.781592    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" (UID: "4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:15.781672 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.781618    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" (UID: "4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:15.781672 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.781640    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" (UID: "4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:15.783316 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.783288    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" (UID: "4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:15.783462 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.783437    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kube-api-access-8vcn7" (OuterVolumeSpecName: "kube-api-access-8vcn7") pod "4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" (UID: "4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88"). InnerVolumeSpecName "kube-api-access-8vcn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:15.882458 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.882415    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:24:15.882458 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.882447    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:24:15.882458 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.882457    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vcn7\" (UniqueName: \"kubernetes.io/projected/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kube-api-access-8vcn7\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:24:15.882458 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.882466    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:24:15.882788 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:15.882476    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:24:16.323726 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.323691    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerID="96f2519804a14ac781903bca3d54392eb1a70d3d75a7bf8a06d8b6dbcdaa5275" exitCode=0
Sep 16 19:24:16.323921 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.323761    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"
Sep 16 19:24:16.323921 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.323776    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" event={"ID":"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88","Type":"ContainerDied","Data":"96f2519804a14ac781903bca3d54392eb1a70d3d75a7bf8a06d8b6dbcdaa5275"}
Sep 16 19:24:16.323921 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.323864    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg" event={"ID":"4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88","Type":"ContainerDied","Data":"e8152896df4fd78347ba977ee7f3bbdc84d5f494fd5c4d95ddfa6b600a7eae87"}
Sep 16 19:24:16.323921 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.323892    2552 scope.go:117] "RemoveContainer" containerID="3dfb6b3b8852c7db098b3e9da6e75a1ee644d5df9f1eebd76dd78dd2e694e51a"
Sep 16 19:24:16.332873 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.332856    2552 scope.go:117] "RemoveContainer" containerID="96f2519804a14ac781903bca3d54392eb1a70d3d75a7bf8a06d8b6dbcdaa5275"
Sep 16 19:24:16.340539 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.340520    2552 scope.go:117] "RemoveContainer" containerID="e299b5dd7e10632332b17e719de3578f071bcce688952753e99645aa9d15e029"
Sep 16 19:24:16.347841 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.347823    2552 scope.go:117] "RemoveContainer" containerID="3dfb6b3b8852c7db098b3e9da6e75a1ee644d5df9f1eebd76dd78dd2e694e51a"
Sep 16 19:24:16.348175 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:24:16.348144    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dfb6b3b8852c7db098b3e9da6e75a1ee644d5df9f1eebd76dd78dd2e694e51a\": container with ID starting with 3dfb6b3b8852c7db098b3e9da6e75a1ee644d5df9f1eebd76dd78dd2e694e51a not found: ID does not exist" containerID="3dfb6b3b8852c7db098b3e9da6e75a1ee644d5df9f1eebd76dd78dd2e694e51a"
Sep 16 19:24:16.348266 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.348185    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfb6b3b8852c7db098b3e9da6e75a1ee644d5df9f1eebd76dd78dd2e694e51a"} err="failed to get container status \"3dfb6b3b8852c7db098b3e9da6e75a1ee644d5df9f1eebd76dd78dd2e694e51a\": rpc error: code = NotFound desc = could not find container \"3dfb6b3b8852c7db098b3e9da6e75a1ee644d5df9f1eebd76dd78dd2e694e51a\": container with ID starting with 3dfb6b3b8852c7db098b3e9da6e75a1ee644d5df9f1eebd76dd78dd2e694e51a not found: ID does not exist"
Sep 16 19:24:16.348266 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.348207    2552 scope.go:117] "RemoveContainer" containerID="96f2519804a14ac781903bca3d54392eb1a70d3d75a7bf8a06d8b6dbcdaa5275"
Sep 16 19:24:16.348473 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:24:16.348456    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f2519804a14ac781903bca3d54392eb1a70d3d75a7bf8a06d8b6dbcdaa5275\": container with ID starting with 96f2519804a14ac781903bca3d54392eb1a70d3d75a7bf8a06d8b6dbcdaa5275 not found: ID does not exist" containerID="96f2519804a14ac781903bca3d54392eb1a70d3d75a7bf8a06d8b6dbcdaa5275"
Sep 16 19:24:16.348509 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.348480    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f2519804a14ac781903bca3d54392eb1a70d3d75a7bf8a06d8b6dbcdaa5275"} err="failed to get container status \"96f2519804a14ac781903bca3d54392eb1a70d3d75a7bf8a06d8b6dbcdaa5275\": rpc error: code = NotFound desc = could not find container \"96f2519804a14ac781903bca3d54392eb1a70d3d75a7bf8a06d8b6dbcdaa5275\": container with ID starting with 96f2519804a14ac781903bca3d54392eb1a70d3d75a7bf8a06d8b6dbcdaa5275 not found: ID does not exist"
Sep 16 19:24:16.348546 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.348508    2552 scope.go:117] "RemoveContainer" containerID="e299b5dd7e10632332b17e719de3578f071bcce688952753e99645aa9d15e029"
Sep 16 19:24:16.348771 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:24:16.348752    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e299b5dd7e10632332b17e719de3578f071bcce688952753e99645aa9d15e029\": container with ID starting with e299b5dd7e10632332b17e719de3578f071bcce688952753e99645aa9d15e029 not found: ID does not exist" containerID="e299b5dd7e10632332b17e719de3578f071bcce688952753e99645aa9d15e029"
Sep 16 19:24:16.348812 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.348783    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e299b5dd7e10632332b17e719de3578f071bcce688952753e99645aa9d15e029"} err="failed to get container status \"e299b5dd7e10632332b17e719de3578f071bcce688952753e99645aa9d15e029\": rpc error: code = NotFound desc = could not find container \"e299b5dd7e10632332b17e719de3578f071bcce688952753e99645aa9d15e029\": container with ID starting with e299b5dd7e10632332b17e719de3578f071bcce688952753e99645aa9d15e029 not found: ID does not exist"
Sep 16 19:24:16.351579 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.351557    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"]
Sep 16 19:24:16.354283 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.354263    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg"]
Sep 16 19:24:16.626684 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.626634    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" path="/var/lib/kubelet/pods/4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88/volumes"
Sep 16 19:24:16.956963 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.956880    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"]
Sep 16 19:24:16.957356 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.957223    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="storage-initializer"
Sep 16 19:24:16.957356 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.957235    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="storage-initializer"
Sep 16 19:24:16.957356 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.957253    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kube-rbac-proxy"
Sep 16 19:24:16.957356 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.957259    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kube-rbac-proxy"
Sep 16 19:24:16.957356 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.957266    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kserve-container"
Sep 16 19:24:16.957356 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.957272    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kserve-container"
Sep 16 19:24:16.957356 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.957327    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kserve-container"
Sep 16 19:24:16.957356 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.957335    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d0af7ef-7edc-4d3a-b525-71a5a8cf9d88" containerName="kube-rbac-proxy"
Sep 16 19:24:16.960898 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.960879    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:16.963040 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.963015    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:24:16.963223 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.963203    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 16 19:24:16.963288 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.963244    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 16 19:24:16.971273 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:16.971240    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"]
Sep 16 19:24:17.091986 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:17.091934    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-5584f7c89b-l4jlw\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:17.091986 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:17.091985    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9nd2\" (UniqueName: \"kubernetes.io/projected/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kube-api-access-h9nd2\") pod \"isvc-sklearn-s3-predictor-5584f7c89b-l4jlw\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:17.092256 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:17.092007    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-5584f7c89b-l4jlw\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:17.092256 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:17.092136    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5584f7c89b-l4jlw\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:17.192745 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:17.192690    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9nd2\" (UniqueName: \"kubernetes.io/projected/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kube-api-access-h9nd2\") pod \"isvc-sklearn-s3-predictor-5584f7c89b-l4jlw\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:17.192745 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:17.192741    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-5584f7c89b-l4jlw\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:17.193031 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:17.192787    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5584f7c89b-l4jlw\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:17.193031 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:17.192874    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-5584f7c89b-l4jlw\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:17.193308 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:17.193282    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-5584f7c89b-l4jlw\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:17.193716 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:17.193680    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-5584f7c89b-l4jlw\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:17.195932 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:17.195897    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5584f7c89b-l4jlw\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:17.216373 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:17.216290    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9nd2\" (UniqueName: \"kubernetes.io/projected/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kube-api-access-h9nd2\") pod \"isvc-sklearn-s3-predictor-5584f7c89b-l4jlw\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:17.272324 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:17.272285    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:17.555819 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:17.555734    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"]
Sep 16 19:24:17.559033 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:24:17.558995    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfbf6f0d_e50e_4a76_832f_c0d2c8c9e03a.slice/crio-b14f8ba5e3eea942de74c60a182b26cc09ec956cdb56e9993a337e1beffb4b7e WatchSource:0}: Error finding container b14f8ba5e3eea942de74c60a182b26cc09ec956cdb56e9993a337e1beffb4b7e: Status 404 returned error can't find the container with id b14f8ba5e3eea942de74c60a182b26cc09ec956cdb56e9993a337e1beffb4b7e
Sep 16 19:24:17.620296 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:17.620274    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:24:17.620504 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:24:17.620482    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:24:18.339035 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:18.338992    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" event={"ID":"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a","Type":"ContainerStarted","Data":"f2314cf272c5de9af47d7e6affab05ee99567a946ac23e9b28ecebd9704eecd8"}
Sep 16 19:24:18.339452 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:18.339070    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" event={"ID":"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a","Type":"ContainerStarted","Data":"b14f8ba5e3eea942de74c60a182b26cc09ec956cdb56e9993a337e1beffb4b7e"}
Sep 16 19:24:19.180456 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:19.180419    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"]
Sep 16 19:24:19.180690 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:19.180661    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8" podUID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerName="kserve-container" containerID="cri-o://8f5b3ea0db8dd9e90e84f3860c29350e9547ff3b824f6a5658af78a03b1e0c35" gracePeriod=30
Sep 16 19:24:19.180772 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:19.180698    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8" podUID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerName="kube-rbac-proxy" containerID="cri-o://3004497a128ed7a019967ce276fde339f518d317c2b90070667f39f84e93461c" gracePeriod=30
Sep 16 19:24:19.345753 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:19.345720    2552 generic.go:356] "Generic (PLEG): container finished" podID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerID="3004497a128ed7a019967ce276fde339f518d317c2b90070667f39f84e93461c" exitCode=2
Sep 16 19:24:19.346204 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:19.345788    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8" event={"ID":"04b1116b-a7db-461f-b4fd-77c74c3fb2fd","Type":"ContainerDied","Data":"3004497a128ed7a019967ce276fde339f518d317c2b90070667f39f84e93461c"}
Sep 16 19:24:19.347341 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:19.347314    2552 generic.go:356] "Generic (PLEG): container finished" podID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerID="f2314cf272c5de9af47d7e6affab05ee99567a946ac23e9b28ecebd9704eecd8" exitCode=0
Sep 16 19:24:19.347485 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:19.347401    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" event={"ID":"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a","Type":"ContainerDied","Data":"f2314cf272c5de9af47d7e6affab05ee99567a946ac23e9b28ecebd9704eecd8"}
Sep 16 19:24:19.939946 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:19.939918    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:24:20.122672 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.122644    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kube-rbac-proxy-sar-config\") pod \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") "
Sep 16 19:24:20.122841 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.122693    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-cabundle-cert\") pod \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") "
Sep 16 19:24:20.122841 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.122758    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-proxy-tls\") pod \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") "
Sep 16 19:24:20.122841 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.122789    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kserve-provision-location\") pod \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") "
Sep 16 19:24:20.122841 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.122815    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7z6g\" (UniqueName: \"kubernetes.io/projected/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kube-api-access-n7z6g\") pod \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\" (UID: \"04b1116b-a7db-461f-b4fd-77c74c3fb2fd\") "
Sep 16 19:24:20.123158 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.123136    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "04b1116b-a7db-461f-b4fd-77c74c3fb2fd" (UID: "04b1116b-a7db-461f-b4fd-77c74c3fb2fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:20.123158 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.123146    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "04b1116b-a7db-461f-b4fd-77c74c3fb2fd" (UID: "04b1116b-a7db-461f-b4fd-77c74c3fb2fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:20.123296 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.123146    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "04b1116b-a7db-461f-b4fd-77c74c3fb2fd" (UID: "04b1116b-a7db-461f-b4fd-77c74c3fb2fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:20.124969 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.124949    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "04b1116b-a7db-461f-b4fd-77c74c3fb2fd" (UID: "04b1116b-a7db-461f-b4fd-77c74c3fb2fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:20.125039 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.125000    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kube-api-access-n7z6g" (OuterVolumeSpecName: "kube-api-access-n7z6g") pod "04b1116b-a7db-461f-b4fd-77c74c3fb2fd" (UID: "04b1116b-a7db-461f-b4fd-77c74c3fb2fd"). InnerVolumeSpecName "kube-api-access-n7z6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:20.223901 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.223859    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:24:20.223901 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.223894    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:24:20.223901 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.223905    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:24:20.223901 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.223915    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:24:20.224164 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.223923    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7z6g\" (UniqueName: \"kubernetes.io/projected/04b1116b-a7db-461f-b4fd-77c74c3fb2fd-kube-api-access-n7z6g\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:24:20.354061 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.354015    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" event={"ID":"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a","Type":"ContainerStarted","Data":"fcd92dd50480df26fe72383d1ae88b7c2de69a36e87cc5c1b0ac631bada772b4"}
Sep 16 19:24:20.354537 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.354114    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:20.354537 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.354150    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:20.354537 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.354178    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" event={"ID":"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a","Type":"ContainerStarted","Data":"fc2ef692d99ace3fbb25541cd9b7f2b28960d82925006ac51456d90f196452e6"}
Sep 16 19:24:20.355992 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.355966    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 19:24:20.356215 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.356193    2552 generic.go:356] "Generic (PLEG): container finished" podID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerID="8f5b3ea0db8dd9e90e84f3860c29350e9547ff3b824f6a5658af78a03b1e0c35" exitCode=0
Sep 16 19:24:20.356282 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.356247    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"
Sep 16 19:24:20.356362 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.356252    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8" event={"ID":"04b1116b-a7db-461f-b4fd-77c74c3fb2fd","Type":"ContainerDied","Data":"8f5b3ea0db8dd9e90e84f3860c29350e9547ff3b824f6a5658af78a03b1e0c35"}
Sep 16 19:24:20.356413 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.356400    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8" event={"ID":"04b1116b-a7db-461f-b4fd-77c74c3fb2fd","Type":"ContainerDied","Data":"92b6502484a4af3787b784b88c0cb4eea78364ffb5f1338c4d094bd32fc982e3"}
Sep 16 19:24:20.356447 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.356433    2552 scope.go:117] "RemoveContainer" containerID="3004497a128ed7a019967ce276fde339f518d317c2b90070667f39f84e93461c"
Sep 16 19:24:20.371037 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.371013    2552 scope.go:117] "RemoveContainer" containerID="8f5b3ea0db8dd9e90e84f3860c29350e9547ff3b824f6a5658af78a03b1e0c35"
Sep 16 19:24:20.373253 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.373165    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" podStartSLOduration=3.003335599 podStartE2EDuration="4.373144598s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.369808999s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:20.373000502 +0000 UTC m=+2656.267187774" watchObservedRunningTime="2026-09-16 19:24:20.373144598 +0000 UTC m=+2656.267331868"
Sep 16 19:24:20.380079 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.380056    2552 scope.go:117] "RemoveContainer" containerID="6b763fcda43010bdda9d884258ca76138c8103f56785a428e0df1da60f64d4e6"
Sep 16 19:24:20.388726 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.388669    2552 scope.go:117] "RemoveContainer" containerID="3004497a128ed7a019967ce276fde339f518d317c2b90070667f39f84e93461c"
Sep 16 19:24:20.389144 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:24:20.389115    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3004497a128ed7a019967ce276fde339f518d317c2b90070667f39f84e93461c\": container with ID starting with 3004497a128ed7a019967ce276fde339f518d317c2b90070667f39f84e93461c not found: ID does not exist" containerID="3004497a128ed7a019967ce276fde339f518d317c2b90070667f39f84e93461c"
Sep 16 19:24:20.389239 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.389155    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3004497a128ed7a019967ce276fde339f518d317c2b90070667f39f84e93461c"} err="failed to get container status \"3004497a128ed7a019967ce276fde339f518d317c2b90070667f39f84e93461c\": rpc error: code = NotFound desc = could not find container \"3004497a128ed7a019967ce276fde339f518d317c2b90070667f39f84e93461c\": container with ID starting with 3004497a128ed7a019967ce276fde339f518d317c2b90070667f39f84e93461c not found: ID does not exist"
Sep 16 19:24:20.389239 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.389178    2552 scope.go:117] "RemoveContainer" containerID="8f5b3ea0db8dd9e90e84f3860c29350e9547ff3b824f6a5658af78a03b1e0c35"
Sep 16 19:24:20.389511 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:24:20.389481    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5b3ea0db8dd9e90e84f3860c29350e9547ff3b824f6a5658af78a03b1e0c35\": container with ID starting with 8f5b3ea0db8dd9e90e84f3860c29350e9547ff3b824f6a5658af78a03b1e0c35 not found: ID does not exist" containerID="8f5b3ea0db8dd9e90e84f3860c29350e9547ff3b824f6a5658af78a03b1e0c35"
Sep 16 19:24:20.389620 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.389518    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5b3ea0db8dd9e90e84f3860c29350e9547ff3b824f6a5658af78a03b1e0c35"} err="failed to get container status \"8f5b3ea0db8dd9e90e84f3860c29350e9547ff3b824f6a5658af78a03b1e0c35\": rpc error: code = NotFound desc = could not find container \"8f5b3ea0db8dd9e90e84f3860c29350e9547ff3b824f6a5658af78a03b1e0c35\": container with ID starting with 8f5b3ea0db8dd9e90e84f3860c29350e9547ff3b824f6a5658af78a03b1e0c35 not found: ID does not exist"
Sep 16 19:24:20.389620 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.389537    2552 scope.go:117] "RemoveContainer" containerID="6b763fcda43010bdda9d884258ca76138c8103f56785a428e0df1da60f64d4e6"
Sep 16 19:24:20.389935 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:24:20.389906    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b763fcda43010bdda9d884258ca76138c8103f56785a428e0df1da60f64d4e6\": container with ID starting with 6b763fcda43010bdda9d884258ca76138c8103f56785a428e0df1da60f64d4e6 not found: ID does not exist" containerID="6b763fcda43010bdda9d884258ca76138c8103f56785a428e0df1da60f64d4e6"
Sep 16 19:24:20.390036 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.389933    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b763fcda43010bdda9d884258ca76138c8103f56785a428e0df1da60f64d4e6"} err="failed to get container status \"6b763fcda43010bdda9d884258ca76138c8103f56785a428e0df1da60f64d4e6\": rpc error: code = NotFound desc = could not find container \"6b763fcda43010bdda9d884258ca76138c8103f56785a428e0df1da60f64d4e6\": container with ID starting with 6b763fcda43010bdda9d884258ca76138c8103f56785a428e0df1da60f64d4e6 not found: ID does not exist"
Sep 16 19:24:20.390435 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.390389    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"]
Sep 16 19:24:20.392181 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.392157    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8"]
Sep 16 19:24:20.625824 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:20.625737    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" path="/var/lib/kubelet/pods/04b1116b-a7db-461f-b4fd-77c74c3fb2fd/volumes"
Sep 16 19:24:21.361831 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:21.361786    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 19:24:24.329283 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.329223    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"]
Sep 16 19:24:24.329749 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.329556    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerName="storage-initializer"
Sep 16 19:24:24.329749 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.329568    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerName="storage-initializer"
Sep 16 19:24:24.329749 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.329591    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerName="kube-rbac-proxy"
Sep 16 19:24:24.329749 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.329596    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerName="kube-rbac-proxy"
Sep 16 19:24:24.329749 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.329606    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerName="kserve-container"
Sep 16 19:24:24.329749 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.329611    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerName="kserve-container"
Sep 16 19:24:24.329749 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.329662    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerName="kserve-container"
Sep 16 19:24:24.329749 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.329673    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b1116b-a7db-461f-b4fd-77c74c3fb2fd" containerName="kube-rbac-proxy"
Sep 16 19:24:24.332963 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.332943    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:24.335369 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.335331    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 16 19:24:24.335369 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.335360    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:24:24.335613 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.335361    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:24:24.335613 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.335340    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:24:24.342946 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.342913    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"]
Sep 16 19:24:24.460002 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.459956    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:24.460216 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.460029    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:24.460216 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.460060    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h59qg\" (UniqueName: \"kubernetes.io/projected/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kube-api-access-h59qg\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:24.460216 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.460080    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:24.460216 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.460166    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:24.561223 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.561182    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:24.561398 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.561234    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h59qg\" (UniqueName: \"kubernetes.io/projected/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kube-api-access-h59qg\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:24.561398 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.561343    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:24.561398 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.561386    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:24.561569 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:24:24.561462    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 16 19:24:24.561569 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.561484    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:24.561569 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:24:24.561533    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-proxy-tls podName:2c0e04e7-09b5-4177-b42f-289c4f2f9b2a nodeName:}" failed. No retries permitted until 2026-09-16 19:24:25.061515982 +0000 UTC m=+2660.955703229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg" (UID: "2c0e04e7-09b5-4177-b42f-289c4f2f9b2a") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 16 19:24:24.561843 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.561819    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:24.561980 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.561963    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:24.562110 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.562065    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:24.572337 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:24.572310    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h59qg\" (UniqueName: \"kubernetes.io/projected/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kube-api-access-h59qg\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:25.064881 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:25.064829    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:25.067454 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:25.067433    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:25.244804 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:25.244757    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:25.519299 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:25.519272    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"]
Sep 16 19:24:25.521812 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:24:25.521785    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0e04e7_09b5_4177_b42f_289c4f2f9b2a.slice/crio-bcbc51936b4a5705515a4e33df0c83ce4f08e26f5e10516ecf40fc0d4c143368 WatchSource:0}: Error finding container bcbc51936b4a5705515a4e33df0c83ce4f08e26f5e10516ecf40fc0d4c143368: Status 404 returned error can't find the container with id bcbc51936b4a5705515a4e33df0c83ce4f08e26f5e10516ecf40fc0d4c143368
Sep 16 19:24:26.365963 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:26.365935    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:24:26.366448 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:26.366420    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 19:24:26.384470 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:26.384437    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg" event={"ID":"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a","Type":"ContainerStarted","Data":"7ae2fbfb6bcb88fcd0f64e25da4472c6673b363b500b9b03865056187334ee96"}
Sep 16 19:24:26.384599 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:26.384501    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg" event={"ID":"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a","Type":"ContainerStarted","Data":"bcbc51936b4a5705515a4e33df0c83ce4f08e26f5e10516ecf40fc0d4c143368"}
Sep 16 19:24:30.404420 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:30.404389    2552 generic.go:356] "Generic (PLEG): container finished" podID="2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" containerID="7ae2fbfb6bcb88fcd0f64e25da4472c6673b363b500b9b03865056187334ee96" exitCode=0
Sep 16 19:24:30.404807 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:30.404460    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg" event={"ID":"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a","Type":"ContainerDied","Data":"7ae2fbfb6bcb88fcd0f64e25da4472c6673b363b500b9b03865056187334ee96"}
Sep 16 19:24:31.411454 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:31.411407    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg" event={"ID":"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a","Type":"ContainerStarted","Data":"c15bd60158389f1979efb8a1c653c71f8233db63c617e5806e3c7f26a7a564c8"}
Sep 16 19:24:31.411890 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:31.411495    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg" event={"ID":"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a","Type":"ContainerStarted","Data":"9463005a3c20f9aa0e2258b88dc6d6bedffa99290d3aff5905da38a35ed9e4e5"}
Sep 16 19:24:31.411890 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:31.411724    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:31.436001 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:31.435929    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg" podStartSLOduration=3.216835135 podStartE2EDuration="7.435910642s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.219075507s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:31.433562746 +0000 UTC m=+2667.327750040" watchObservedRunningTime="2026-09-16 19:24:31.435910642 +0000 UTC m=+2667.330097910"
Sep 16 19:24:32.417080 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:32.417047    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:32.619723 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:32.619693    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:24:32.619898 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:24:32.619844    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:24:36.367305 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:36.367262    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 19:24:38.429934 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:38.429900    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:24:43.619547 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:43.619510    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:24:43.620036 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:24:43.619700    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:24:46.366716 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:46.366676    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 19:24:54.622499 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:54.622469    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:24:54.622924 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:24:54.622648    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:24:56.366462 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:24:56.366377    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 19:25:05.620267 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:05.620235    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:25:05.620668 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:25:05.620419    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:25:06.367168 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:06.367127    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 19:25:08.434075 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:08.434042    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:25:14.407377 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:14.407341    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"]
Sep 16 19:25:14.407980 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:14.407606    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg" podUID="2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" containerName="kserve-container" containerID="cri-o://9463005a3c20f9aa0e2258b88dc6d6bedffa99290d3aff5905da38a35ed9e4e5" gracePeriod=30
Sep 16 19:25:14.407980 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:14.407626    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg" podUID="2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" containerName="kube-rbac-proxy" containerID="cri-o://c15bd60158389f1979efb8a1c653c71f8233db63c617e5806e3c7f26a7a564c8" gracePeriod=30
Sep 16 19:25:14.622664 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:14.622632    2552 generic.go:356] "Generic (PLEG): container finished" podID="2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" containerID="c15bd60158389f1979efb8a1c653c71f8233db63c617e5806e3c7f26a7a564c8" exitCode=2
Sep 16 19:25:14.625804 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:14.625776    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg" event={"ID":"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a","Type":"ContainerDied","Data":"c15bd60158389f1979efb8a1c653c71f8233db63c617e5806e3c7f26a7a564c8"}
Sep 16 19:25:15.630557 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:15.630523    2552 generic.go:356] "Generic (PLEG): container finished" podID="2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" containerID="9463005a3c20f9aa0e2258b88dc6d6bedffa99290d3aff5905da38a35ed9e4e5" exitCode=0
Sep 16 19:25:15.630959 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:15.630598    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg" event={"ID":"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a","Type":"ContainerDied","Data":"9463005a3c20f9aa0e2258b88dc6d6bedffa99290d3aff5905da38a35ed9e4e5"}
Sep 16 19:25:15.758183 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:15.758148    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:25:15.907291 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:15.907255    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kube-rbac-proxy-sar-config\") pod \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") "
Sep 16 19:25:15.907457 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:15.907303    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-cabundle-cert\") pod \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") "
Sep 16 19:25:15.907457 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:15.907336    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-proxy-tls\") pod \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") "
Sep 16 19:25:15.907457 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:15.907375    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kserve-provision-location\") pod \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") "
Sep 16 19:25:15.907622 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:15.907459    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h59qg\" (UniqueName: \"kubernetes.io/projected/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kube-api-access-h59qg\") pod \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\" (UID: \"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a\") "
Sep 16 19:25:15.907707 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:15.907681    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" (UID: "2c0e04e7-09b5-4177-b42f-289c4f2f9b2a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:15.907775 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:15.907715    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" (UID: "2c0e04e7-09b5-4177-b42f-289c4f2f9b2a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:15.907834 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:15.907814    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" (UID: "2c0e04e7-09b5-4177-b42f-289c4f2f9b2a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:15.909523 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:15.909501    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kube-api-access-h59qg" (OuterVolumeSpecName: "kube-api-access-h59qg") pod "2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" (UID: "2c0e04e7-09b5-4177-b42f-289c4f2f9b2a"). InnerVolumeSpecName "kube-api-access-h59qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:15.909616 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:15.909589    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" (UID: "2c0e04e7-09b5-4177-b42f-289c4f2f9b2a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:16.009072 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:16.009019    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:25:16.009072 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:16.009065    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h59qg\" (UniqueName: \"kubernetes.io/projected/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kube-api-access-h59qg\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:25:16.009072 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:16.009076    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:25:16.009072 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:16.009108    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:25:16.009410 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:16.009121    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:25:16.366534 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:16.366433    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 19:25:16.637192 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:16.637162    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg" event={"ID":"2c0e04e7-09b5-4177-b42f-289c4f2f9b2a","Type":"ContainerDied","Data":"bcbc51936b4a5705515a4e33df0c83ce4f08e26f5e10516ecf40fc0d4c143368"}
Sep 16 19:25:16.637514 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:16.637219    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"
Sep 16 19:25:16.637514 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:16.637230    2552 scope.go:117] "RemoveContainer" containerID="c15bd60158389f1979efb8a1c653c71f8233db63c617e5806e3c7f26a7a564c8"
Sep 16 19:25:16.645827 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:16.645805    2552 scope.go:117] "RemoveContainer" containerID="9463005a3c20f9aa0e2258b88dc6d6bedffa99290d3aff5905da38a35ed9e4e5"
Sep 16 19:25:16.653963 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:16.653939    2552 scope.go:117] "RemoveContainer" containerID="7ae2fbfb6bcb88fcd0f64e25da4472c6673b363b500b9b03865056187334ee96"
Sep 16 19:25:16.657751 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:16.657724    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"]
Sep 16 19:25:16.665717 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:16.665688    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg"]
Sep 16 19:25:18.625874 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:18.625827    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" path="/var/lib/kubelet/pods/2c0e04e7-09b5-4177-b42f-289c4f2f9b2a/volumes"
Sep 16 19:25:20.619637 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:20.619602    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:25:20.620036 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:25:20.619784    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:25:26.367225 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:26.367183    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 19:25:32.620190 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:32.620152    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:25:32.620725 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:25:32.620335    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:25:36.367111 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:36.367055    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:25:37.033284 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.033247    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"]
Sep 16 19:25:37.033513 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.033470    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kserve-container" containerID="cri-o://fc2ef692d99ace3fbb25541cd9b7f2b28960d82925006ac51456d90f196452e6" gracePeriod=30
Sep 16 19:25:37.033589 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.033531    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kube-rbac-proxy" containerID="cri-o://fcd92dd50480df26fe72383d1ae88b7c2de69a36e87cc5c1b0ac631bada772b4" gracePeriod=30
Sep 16 19:25:37.210640 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.210603    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"]
Sep 16 19:25:37.210971 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.210931    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" containerName="kube-rbac-proxy"
Sep 16 19:25:37.210971 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.210947    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" containerName="kube-rbac-proxy"
Sep 16 19:25:37.211062 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.210984    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" containerName="kserve-container"
Sep 16 19:25:37.211062 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.210993    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" containerName="kserve-container"
Sep 16 19:25:37.211062 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.211011    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" containerName="storage-initializer"
Sep 16 19:25:37.211062 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.211019    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" containerName="storage-initializer"
Sep 16 19:25:37.211205 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.211075    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" containerName="kube-rbac-proxy"
Sep 16 19:25:37.211205 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.211104    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c0e04e7-09b5-4177-b42f-289c4f2f9b2a" containerName="kserve-container"
Sep 16 19:25:37.214188 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.214168    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.217020 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.216987    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 16 19:25:37.217179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.216987    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:25:37.217179 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.216988    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 16 19:25:37.227962 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.227930    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"]
Sep 16 19:25:37.291638 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.291555    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7x8\" (UniqueName: \"kubernetes.io/projected/e033c16d-de7b-4859-8ee3-6aa647df896c-kube-api-access-zm7x8\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.291638 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.291602    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e033c16d-de7b-4859-8ee3-6aa647df896c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.291638 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.291638    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e033c16d-de7b-4859-8ee3-6aa647df896c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.291900 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.291716    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e033c16d-de7b-4859-8ee3-6aa647df896c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.291900 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.291753    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e033c16d-de7b-4859-8ee3-6aa647df896c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.392760 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.392714    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7x8\" (UniqueName: \"kubernetes.io/projected/e033c16d-de7b-4859-8ee3-6aa647df896c-kube-api-access-zm7x8\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.393276 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.392769    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e033c16d-de7b-4859-8ee3-6aa647df896c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.393276 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.392805    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e033c16d-de7b-4859-8ee3-6aa647df896c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.393276 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.392857    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e033c16d-de7b-4859-8ee3-6aa647df896c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.393276 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.392893    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e033c16d-de7b-4859-8ee3-6aa647df896c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.393524 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.393306    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e033c16d-de7b-4859-8ee3-6aa647df896c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.393616 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.393593    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e033c16d-de7b-4859-8ee3-6aa647df896c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.393749 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.393727    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e033c16d-de7b-4859-8ee3-6aa647df896c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.395398 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.395382    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e033c16d-de7b-4859-8ee3-6aa647df896c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.405789 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.405768    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7x8\" (UniqueName: \"kubernetes.io/projected/e033c16d-de7b-4859-8ee3-6aa647df896c-kube-api-access-zm7x8\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.524699 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.524659    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:37.740532 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.740499    2552 generic.go:356] "Generic (PLEG): container finished" podID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerID="fcd92dd50480df26fe72383d1ae88b7c2de69a36e87cc5c1b0ac631bada772b4" exitCode=2
Sep 16 19:25:37.740709 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.740552    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" event={"ID":"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a","Type":"ContainerDied","Data":"fcd92dd50480df26fe72383d1ae88b7c2de69a36e87cc5c1b0ac631bada772b4"}
Sep 16 19:25:37.803101 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:37.803061    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"]
Sep 16 19:25:37.805604 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:25:37.805570    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode033c16d_de7b_4859_8ee3_6aa647df896c.slice/crio-ba9312a55f2cebada6ebefc20c8dda342017c0fc305fbaf1bc0191b5d6b97f87 WatchSource:0}: Error finding container ba9312a55f2cebada6ebefc20c8dda342017c0fc305fbaf1bc0191b5d6b97f87: Status 404 returned error can't find the container with id ba9312a55f2cebada6ebefc20c8dda342017c0fc305fbaf1bc0191b5d6b97f87
Sep 16 19:25:38.746463 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:38.746422    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" event={"ID":"e033c16d-de7b-4859-8ee3-6aa647df896c","Type":"ContainerStarted","Data":"6832d2947eafe454da3807bb1fa596e3d9f7735537e359b9b01bcfd5f0ecd3b2"}
Sep 16 19:25:38.746928 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:38.746498    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" event={"ID":"e033c16d-de7b-4859-8ee3-6aa647df896c","Type":"ContainerStarted","Data":"ba9312a55f2cebada6ebefc20c8dda342017c0fc305fbaf1bc0191b5d6b97f87"}
Sep 16 19:25:39.752509 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:39.752463    2552 generic.go:356] "Generic (PLEG): container finished" podID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerID="6832d2947eafe454da3807bb1fa596e3d9f7735537e359b9b01bcfd5f0ecd3b2" exitCode=0
Sep 16 19:25:39.752925 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:39.752547    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" event={"ID":"e033c16d-de7b-4859-8ee3-6aa647df896c","Type":"ContainerDied","Data":"6832d2947eafe454da3807bb1fa596e3d9f7735537e359b9b01bcfd5f0ecd3b2"}
Sep 16 19:25:40.760019 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:40.759980    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" event={"ID":"e033c16d-de7b-4859-8ee3-6aa647df896c","Type":"ContainerStarted","Data":"3325f1c62593adfd209a137f448be0393bdd84e3cf0ff568e8d405ce98ed2a94"}
Sep 16 19:25:40.760455 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:40.760054    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" event={"ID":"e033c16d-de7b-4859-8ee3-6aa647df896c","Type":"ContainerStarted","Data":"d4ab4c1991604dec2244c1eb7af34f5d69eb0b2d52b3e22a3a33fce620ccad44"}
Sep 16 19:25:40.760455 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:40.760140    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:40.782001 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:40.781941    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" podStartSLOduration=2.564174498 podStartE2EDuration="3.781921084s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.217746586s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:40.779181931 +0000 UTC m=+2736.673369200" watchObservedRunningTime="2026-09-16 19:25:40.781921084 +0000 UTC m=+2736.676108352"
Sep 16 19:25:41.363037 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:41.362991    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 16 19:25:41.764950 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:41.764916    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:41.766416 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:41.766385    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 16 19:25:42.287840 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.287811    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:25:42.437651 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.437566    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kube-rbac-proxy-sar-config\") pod \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") "
Sep 16 19:25:42.437651 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.437607    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kserve-provision-location\") pod \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") "
Sep 16 19:25:42.437651 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.437633    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9nd2\" (UniqueName: \"kubernetes.io/projected/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kube-api-access-h9nd2\") pod \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") "
Sep 16 19:25:42.437971 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.437709    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-proxy-tls\") pod \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\" (UID: \"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a\") "
Sep 16 19:25:42.437971 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.437959    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" (UID: "dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:42.438120 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.438029    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" (UID: "dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:42.439857 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.439820    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" (UID: "dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:42.439992 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.439948    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kube-api-access-h9nd2" (OuterVolumeSpecName: "kube-api-access-h9nd2") pod "dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" (UID: "dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a"). InnerVolumeSpecName "kube-api-access-h9nd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:42.539173 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.539138    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:25:42.539173 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.539166    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:25:42.539173 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.539178    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9nd2\" (UniqueName: \"kubernetes.io/projected/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-kube-api-access-h9nd2\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:25:42.539393 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.539188    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:25:42.770846 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.770752    2552 generic.go:356] "Generic (PLEG): container finished" podID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerID="fc2ef692d99ace3fbb25541cd9b7f2b28960d82925006ac51456d90f196452e6" exitCode=0
Sep 16 19:25:42.770846 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.770812    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"
Sep 16 19:25:42.771384 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.770840    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" event={"ID":"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a","Type":"ContainerDied","Data":"fc2ef692d99ace3fbb25541cd9b7f2b28960d82925006ac51456d90f196452e6"}
Sep 16 19:25:42.771384 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.770918    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw" event={"ID":"dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a","Type":"ContainerDied","Data":"b14f8ba5e3eea942de74c60a182b26cc09ec956cdb56e9993a337e1beffb4b7e"}
Sep 16 19:25:42.771384 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.770958    2552 scope.go:117] "RemoveContainer" containerID="fcd92dd50480df26fe72383d1ae88b7c2de69a36e87cc5c1b0ac631bada772b4"
Sep 16 19:25:42.771627 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.771604    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 16 19:25:42.780466 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.780444    2552 scope.go:117] "RemoveContainer" containerID="fc2ef692d99ace3fbb25541cd9b7f2b28960d82925006ac51456d90f196452e6"
Sep 16 19:25:42.788655 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.788632    2552 scope.go:117] "RemoveContainer" containerID="f2314cf272c5de9af47d7e6affab05ee99567a946ac23e9b28ecebd9704eecd8"
Sep 16 19:25:42.791776 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.791752    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"]
Sep 16 19:25:42.794582 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.794559    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw"]
Sep 16 19:25:42.797474 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.797455    2552 scope.go:117] "RemoveContainer" containerID="fcd92dd50480df26fe72383d1ae88b7c2de69a36e87cc5c1b0ac631bada772b4"
Sep 16 19:25:42.797817 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:25:42.797796    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd92dd50480df26fe72383d1ae88b7c2de69a36e87cc5c1b0ac631bada772b4\": container with ID starting with fcd92dd50480df26fe72383d1ae88b7c2de69a36e87cc5c1b0ac631bada772b4 not found: ID does not exist" containerID="fcd92dd50480df26fe72383d1ae88b7c2de69a36e87cc5c1b0ac631bada772b4"
Sep 16 19:25:42.797892 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.797828    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd92dd50480df26fe72383d1ae88b7c2de69a36e87cc5c1b0ac631bada772b4"} err="failed to get container status \"fcd92dd50480df26fe72383d1ae88b7c2de69a36e87cc5c1b0ac631bada772b4\": rpc error: code = NotFound desc = could not find container \"fcd92dd50480df26fe72383d1ae88b7c2de69a36e87cc5c1b0ac631bada772b4\": container with ID starting with fcd92dd50480df26fe72383d1ae88b7c2de69a36e87cc5c1b0ac631bada772b4 not found: ID does not exist"
Sep 16 19:25:42.797892 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.797857    2552 scope.go:117] "RemoveContainer" containerID="fc2ef692d99ace3fbb25541cd9b7f2b28960d82925006ac51456d90f196452e6"
Sep 16 19:25:42.798263 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:25:42.798241    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc2ef692d99ace3fbb25541cd9b7f2b28960d82925006ac51456d90f196452e6\": container with ID starting with fc2ef692d99ace3fbb25541cd9b7f2b28960d82925006ac51456d90f196452e6 not found: ID does not exist" containerID="fc2ef692d99ace3fbb25541cd9b7f2b28960d82925006ac51456d90f196452e6"
Sep 16 19:25:42.798316 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.798269    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2ef692d99ace3fbb25541cd9b7f2b28960d82925006ac51456d90f196452e6"} err="failed to get container status \"fc2ef692d99ace3fbb25541cd9b7f2b28960d82925006ac51456d90f196452e6\": rpc error: code = NotFound desc = could not find container \"fc2ef692d99ace3fbb25541cd9b7f2b28960d82925006ac51456d90f196452e6\": container with ID starting with fc2ef692d99ace3fbb25541cd9b7f2b28960d82925006ac51456d90f196452e6 not found: ID does not exist"
Sep 16 19:25:42.798316 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.798284    2552 scope.go:117] "RemoveContainer" containerID="f2314cf272c5de9af47d7e6affab05ee99567a946ac23e9b28ecebd9704eecd8"
Sep 16 19:25:42.798547 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:25:42.798530    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2314cf272c5de9af47d7e6affab05ee99567a946ac23e9b28ecebd9704eecd8\": container with ID starting with f2314cf272c5de9af47d7e6affab05ee99567a946ac23e9b28ecebd9704eecd8 not found: ID does not exist" containerID="f2314cf272c5de9af47d7e6affab05ee99567a946ac23e9b28ecebd9704eecd8"
Sep 16 19:25:42.798608 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:42.798555    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2314cf272c5de9af47d7e6affab05ee99567a946ac23e9b28ecebd9704eecd8"} err="failed to get container status \"f2314cf272c5de9af47d7e6affab05ee99567a946ac23e9b28ecebd9704eecd8\": rpc error: code = NotFound desc = could not find container \"f2314cf272c5de9af47d7e6affab05ee99567a946ac23e9b28ecebd9704eecd8\": container with ID starting with f2314cf272c5de9af47d7e6affab05ee99567a946ac23e9b28ecebd9704eecd8 not found: ID does not exist"
Sep 16 19:25:44.625955 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:44.625918    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" path="/var/lib/kubelet/pods/dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a/volumes"
Sep 16 19:25:47.619581 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:47.619548    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:25:47.619980 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:25:47.619758    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:25:47.775854 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:47.775821    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:25:47.776331 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:47.776307    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 16 19:25:57.776344 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:57.776295    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 16 19:25:59.619918 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:25:59.619882    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:25:59.620360 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:25:59.620076    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:26:07.776986 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:07.776942    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 16 19:26:11.619955 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:11.619920    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:26:11.620383 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:26:11.620126    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:26:17.776904 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:17.776860    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 16 19:26:22.620677 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:22.620638    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:26:22.621075 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:26:22.620851    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:26:27.776393 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:27.776301    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 16 19:26:36.620369 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:36.620331    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:26:36.620761 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:26:36.620567    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:26:37.777033 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:37.776996    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 16 19:26:44.847474 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.847436    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"]
Sep 16 19:26:44.847945 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.847770    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="storage-initializer"
Sep 16 19:26:44.847945 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.847782    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="storage-initializer"
Sep 16 19:26:44.847945 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.847802    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kserve-container"
Sep 16 19:26:44.847945 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.847808    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kserve-container"
Sep 16 19:26:44.847945 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.847821    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kube-rbac-proxy"
Sep 16 19:26:44.847945 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.847826    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kube-rbac-proxy"
Sep 16 19:26:44.847945 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.847878    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kube-rbac-proxy"
Sep 16 19:26:44.847945 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.847885    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfbf6f0d-e50e-4a76-832f-c0d2c8c9e03a" containerName="kserve-container"
Sep 16 19:26:44.850976 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.850957    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:44.853298 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.853278    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 16 19:26:44.853387 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.853319    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:26:44.853567 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.853550    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 16 19:26:44.861403 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.861379    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"]
Sep 16 19:26:44.972372 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.972338    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce42c78a-330c-47ea-8a8d-587ab85d96b9-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmhx6\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:44.972554 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.972440    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmhx6\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:44.972554 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.972493    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce42c78a-330c-47ea-8a8d-587ab85d96b9-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmhx6\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:44.972554 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.972518    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxhbf\" (UniqueName: \"kubernetes.io/projected/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kube-api-access-xxhbf\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmhx6\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:44.972554 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:44.972537    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmhx6\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:45.073789 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:45.073732    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce42c78a-330c-47ea-8a8d-587ab85d96b9-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmhx6\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:45.073789 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:45.073787    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhbf\" (UniqueName: \"kubernetes.io/projected/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kube-api-access-xxhbf\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmhx6\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:45.074163 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:45.073918    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmhx6\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:45.074163 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:45.073998    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce42c78a-330c-47ea-8a8d-587ab85d96b9-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmhx6\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:45.074300 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:45.074174    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmhx6\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:45.074532 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:45.074510    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmhx6\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:45.074775 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:45.074752    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmhx6\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:45.074846 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:45.074785    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce42c78a-330c-47ea-8a8d-587ab85d96b9-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmhx6\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:45.076754 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:45.076719    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce42c78a-330c-47ea-8a8d-587ab85d96b9-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmhx6\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:45.084071 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:45.084045    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxhbf\" (UniqueName: \"kubernetes.io/projected/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kube-api-access-xxhbf\") pod \"isvc-paddle-predictor-6d5c9fd4f6-hmhx6\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:45.162079 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:45.162034    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:26:45.436635 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:45.436548    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"]
Sep 16 19:26:45.440150 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:26:45.440117    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce42c78a_330c_47ea_8a8d_587ab85d96b9.slice/crio-cc82ba4590ba7a99b4f03f854c1565a119eec246ce1ffbf7893e7a2062c1e03e WatchSource:0}: Error finding container cc82ba4590ba7a99b4f03f854c1565a119eec246ce1ffbf7893e7a2062c1e03e: Status 404 returned error can't find the container with id cc82ba4590ba7a99b4f03f854c1565a119eec246ce1ffbf7893e7a2062c1e03e
Sep 16 19:26:46.068735 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:46.068690    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" event={"ID":"ce42c78a-330c-47ea-8a8d-587ab85d96b9","Type":"ContainerStarted","Data":"17007fdb3b990e7c0f6e674adac24dc54a506d54e8d306460cdba044414b1f58"}
Sep 16 19:26:46.069208 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:46.068756    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" event={"ID":"ce42c78a-330c-47ea-8a8d-587ab85d96b9","Type":"ContainerStarted","Data":"cc82ba4590ba7a99b4f03f854c1565a119eec246ce1ffbf7893e7a2062c1e03e"}
Sep 16 19:26:47.776744 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:47.776705    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 16 19:26:48.620545 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:48.620510    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:26:48.620774 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:26:48.620700    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:26:50.088275 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:50.088178    2552 generic.go:356] "Generic (PLEG): container finished" podID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerID="17007fdb3b990e7c0f6e674adac24dc54a506d54e8d306460cdba044414b1f58" exitCode=0
Sep 16 19:26:50.088275 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:50.088225    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" event={"ID":"ce42c78a-330c-47ea-8a8d-587ab85d96b9","Type":"ContainerDied","Data":"17007fdb3b990e7c0f6e674adac24dc54a506d54e8d306460cdba044414b1f58"}
Sep 16 19:26:50.089427 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:50.089410    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:26:57.777317 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:26:57.777279    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:27:02.159990 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:02.159902    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" event={"ID":"ce42c78a-330c-47ea-8a8d-587ab85d96b9","Type":"ContainerStarted","Data":"1da890fcca8223ac935f3abec82240b4bb9217befc545e9192c513a067bbd728"}
Sep 16 19:27:02.159990 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:02.159968    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" event={"ID":"ce42c78a-330c-47ea-8a8d-587ab85d96b9","Type":"ContainerStarted","Data":"cb7183a74dc56eae30c116ff930a2cd3728aa39b1f3b634b18797e834577b28d"}
Sep 16 19:27:02.160418 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:02.160238    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:27:02.160418 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:02.160296    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:27:02.161768 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:02.161740    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 16 19:27:02.180487 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:02.180422    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" podStartSLOduration=2.135683827 podStartE2EDuration="18.18040202s" totalImagesPullingTime="11.803227944s" totalInitContainerRuntime="4.241490249s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:02.178124265 +0000 UTC m=+2818.072311534" watchObservedRunningTime="2026-09-16 19:27:02.18040202 +0000 UTC m=+2818.074589290"
Sep 16 19:27:02.620698 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:02.620614    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:27:02.620863 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:27:02.620802    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:27:03.164367 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:03.164329    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 16 19:27:07.227105 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:07.227053    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"]
Sep 16 19:27:07.227550 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:07.227326    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kserve-container" containerID="cri-o://d4ab4c1991604dec2244c1eb7af34f5d69eb0b2d52b3e22a3a33fce620ccad44" gracePeriod=30
Sep 16 19:27:07.227550 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:07.227380    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kube-rbac-proxy" containerID="cri-o://3325f1c62593adfd209a137f448be0393bdd84e3cf0ff568e8d405ce98ed2a94" gracePeriod=30
Sep 16 19:27:07.771374 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:07.771328    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Sep 16 19:27:07.776591 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:07.776566    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 16 19:27:08.169185 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:08.169155    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:27:08.169793 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:08.169763    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 16 19:27:08.190114 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:08.190066    2552 generic.go:356] "Generic (PLEG): container finished" podID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerID="3325f1c62593adfd209a137f448be0393bdd84e3cf0ff568e8d405ce98ed2a94" exitCode=2
Sep 16 19:27:08.190269 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:08.190154    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" event={"ID":"e033c16d-de7b-4859-8ee3-6aa647df896c","Type":"ContainerDied","Data":"3325f1c62593adfd209a137f448be0393bdd84e3cf0ff568e8d405ce98ed2a94"}
Sep 16 19:27:12.676738 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.676708    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:27:12.735788 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.735674    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e033c16d-de7b-4859-8ee3-6aa647df896c-kube-rbac-proxy-sar-config\") pod \"e033c16d-de7b-4859-8ee3-6aa647df896c\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") "
Sep 16 19:27:12.735788 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.735736    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e033c16d-de7b-4859-8ee3-6aa647df896c-cabundle-cert\") pod \"e033c16d-de7b-4859-8ee3-6aa647df896c\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") "
Sep 16 19:27:12.736046 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.735800    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e033c16d-de7b-4859-8ee3-6aa647df896c-proxy-tls\") pod \"e033c16d-de7b-4859-8ee3-6aa647df896c\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") "
Sep 16 19:27:12.736046 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.735836    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm7x8\" (UniqueName: \"kubernetes.io/projected/e033c16d-de7b-4859-8ee3-6aa647df896c-kube-api-access-zm7x8\") pod \"e033c16d-de7b-4859-8ee3-6aa647df896c\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") "
Sep 16 19:27:12.736046 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.735931    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e033c16d-de7b-4859-8ee3-6aa647df896c-kserve-provision-location\") pod \"e033c16d-de7b-4859-8ee3-6aa647df896c\" (UID: \"e033c16d-de7b-4859-8ee3-6aa647df896c\") "
Sep 16 19:27:12.736285 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.736107    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e033c16d-de7b-4859-8ee3-6aa647df896c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e033c16d-de7b-4859-8ee3-6aa647df896c" (UID: "e033c16d-de7b-4859-8ee3-6aa647df896c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:12.736285 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.736238    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e033c16d-de7b-4859-8ee3-6aa647df896c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e033c16d-de7b-4859-8ee3-6aa647df896c" (UID: "e033c16d-de7b-4859-8ee3-6aa647df896c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:12.736457 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.736336    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e033c16d-de7b-4859-8ee3-6aa647df896c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e033c16d-de7b-4859-8ee3-6aa647df896c" (UID: "e033c16d-de7b-4859-8ee3-6aa647df896c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:12.738344 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.738311    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e033c16d-de7b-4859-8ee3-6aa647df896c-kube-api-access-zm7x8" (OuterVolumeSpecName: "kube-api-access-zm7x8") pod "e033c16d-de7b-4859-8ee3-6aa647df896c" (UID: "e033c16d-de7b-4859-8ee3-6aa647df896c"). InnerVolumeSpecName "kube-api-access-zm7x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:12.738885 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.738855    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e033c16d-de7b-4859-8ee3-6aa647df896c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e033c16d-de7b-4859-8ee3-6aa647df896c" (UID: "e033c16d-de7b-4859-8ee3-6aa647df896c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:12.836551 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.836510    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e033c16d-de7b-4859-8ee3-6aa647df896c-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:27:12.836551 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.836543    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm7x8\" (UniqueName: \"kubernetes.io/projected/e033c16d-de7b-4859-8ee3-6aa647df896c-kube-api-access-zm7x8\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:27:12.836551 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.836554    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e033c16d-de7b-4859-8ee3-6aa647df896c-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:27:12.836551 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.836563    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e033c16d-de7b-4859-8ee3-6aa647df896c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:27:12.836814 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:12.836572    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e033c16d-de7b-4859-8ee3-6aa647df896c-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:27:13.215380 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:13.215344    2552 generic.go:356] "Generic (PLEG): container finished" podID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerID="d4ab4c1991604dec2244c1eb7af34f5d69eb0b2d52b3e22a3a33fce620ccad44" exitCode=0
Sep 16 19:27:13.215548 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:13.215397    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"
Sep 16 19:27:13.215548 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:13.215426    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" event={"ID":"e033c16d-de7b-4859-8ee3-6aa647df896c","Type":"ContainerDied","Data":"d4ab4c1991604dec2244c1eb7af34f5d69eb0b2d52b3e22a3a33fce620ccad44"}
Sep 16 19:27:13.215548 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:13.215496    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6" event={"ID":"e033c16d-de7b-4859-8ee3-6aa647df896c","Type":"ContainerDied","Data":"ba9312a55f2cebada6ebefc20c8dda342017c0fc305fbaf1bc0191b5d6b97f87"}
Sep 16 19:27:13.215548 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:13.215525    2552 scope.go:117] "RemoveContainer" containerID="3325f1c62593adfd209a137f448be0393bdd84e3cf0ff568e8d405ce98ed2a94"
Sep 16 19:27:13.225303 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:13.225287    2552 scope.go:117] "RemoveContainer" containerID="d4ab4c1991604dec2244c1eb7af34f5d69eb0b2d52b3e22a3a33fce620ccad44"
Sep 16 19:27:13.232646 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:13.232628    2552 scope.go:117] "RemoveContainer" containerID="6832d2947eafe454da3807bb1fa596e3d9f7735537e359b9b01bcfd5f0ecd3b2"
Sep 16 19:27:13.240192 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:13.240165    2552 scope.go:117] "RemoveContainer" containerID="3325f1c62593adfd209a137f448be0393bdd84e3cf0ff568e8d405ce98ed2a94"
Sep 16 19:27:13.240505 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:27:13.240481    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3325f1c62593adfd209a137f448be0393bdd84e3cf0ff568e8d405ce98ed2a94\": container with ID starting with 3325f1c62593adfd209a137f448be0393bdd84e3cf0ff568e8d405ce98ed2a94 not found: ID does not exist" containerID="3325f1c62593adfd209a137f448be0393bdd84e3cf0ff568e8d405ce98ed2a94"
Sep 16 19:27:13.240572 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:13.240519    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3325f1c62593adfd209a137f448be0393bdd84e3cf0ff568e8d405ce98ed2a94"} err="failed to get container status \"3325f1c62593adfd209a137f448be0393bdd84e3cf0ff568e8d405ce98ed2a94\": rpc error: code = NotFound desc = could not find container \"3325f1c62593adfd209a137f448be0393bdd84e3cf0ff568e8d405ce98ed2a94\": container with ID starting with 3325f1c62593adfd209a137f448be0393bdd84e3cf0ff568e8d405ce98ed2a94 not found: ID does not exist"
Sep 16 19:27:13.240572 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:13.240545    2552 scope.go:117] "RemoveContainer" containerID="d4ab4c1991604dec2244c1eb7af34f5d69eb0b2d52b3e22a3a33fce620ccad44"
Sep 16 19:27:13.240821 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:27:13.240799    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ab4c1991604dec2244c1eb7af34f5d69eb0b2d52b3e22a3a33fce620ccad44\": container with ID starting with d4ab4c1991604dec2244c1eb7af34f5d69eb0b2d52b3e22a3a33fce620ccad44 not found: ID does not exist" containerID="d4ab4c1991604dec2244c1eb7af34f5d69eb0b2d52b3e22a3a33fce620ccad44"
Sep 16 19:27:13.240874 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:13.240832    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ab4c1991604dec2244c1eb7af34f5d69eb0b2d52b3e22a3a33fce620ccad44"} err="failed to get container status \"d4ab4c1991604dec2244c1eb7af34f5d69eb0b2d52b3e22a3a33fce620ccad44\": rpc error: code = NotFound desc = could not find container \"d4ab4c1991604dec2244c1eb7af34f5d69eb0b2d52b3e22a3a33fce620ccad44\": container with ID starting with d4ab4c1991604dec2244c1eb7af34f5d69eb0b2d52b3e22a3a33fce620ccad44 not found: ID does not exist"
Sep 16 19:27:13.240874 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:13.240855    2552 scope.go:117] "RemoveContainer" containerID="6832d2947eafe454da3807bb1fa596e3d9f7735537e359b9b01bcfd5f0ecd3b2"
Sep 16 19:27:13.241169 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:27:13.241149    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6832d2947eafe454da3807bb1fa596e3d9f7735537e359b9b01bcfd5f0ecd3b2\": container with ID starting with 6832d2947eafe454da3807bb1fa596e3d9f7735537e359b9b01bcfd5f0ecd3b2 not found: ID does not exist" containerID="6832d2947eafe454da3807bb1fa596e3d9f7735537e359b9b01bcfd5f0ecd3b2"
Sep 16 19:27:13.241263 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:13.241174    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6832d2947eafe454da3807bb1fa596e3d9f7735537e359b9b01bcfd5f0ecd3b2"} err="failed to get container status \"6832d2947eafe454da3807bb1fa596e3d9f7735537e359b9b01bcfd5f0ecd3b2\": rpc error: code = NotFound desc = could not find container \"6832d2947eafe454da3807bb1fa596e3d9f7735537e359b9b01bcfd5f0ecd3b2\": container with ID starting with 6832d2947eafe454da3807bb1fa596e3d9f7735537e359b9b01bcfd5f0ecd3b2 not found: ID does not exist"
Sep 16 19:27:13.241771 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:13.241757    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"]
Sep 16 19:27:13.247939 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:13.247919    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6"]
Sep 16 19:27:14.625431 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:14.625397    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" path="/var/lib/kubelet/pods/e033c16d-de7b-4859-8ee3-6aa647df896c/volumes"
Sep 16 19:27:15.619961 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:15.619927    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:27:15.620171 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:27:15.620150    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:27:18.169985 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:18.169939    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 16 19:27:28.170591 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:28.170529    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 16 19:27:30.620698 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:30.620658    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:27:30.621118 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:27:30.620910    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:27:38.170525 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:38.170477    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 16 19:27:42.620255 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:42.620208    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:27:42.620752 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:27:42.620438    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:27:48.170207 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:48.170169    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:27:56.081677 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:56.081645    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"]
Sep 16 19:27:56.082372 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:56.082336    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kserve-container" containerID="cri-o://cb7183a74dc56eae30c116ff930a2cd3728aa39b1f3b634b18797e834577b28d" gracePeriod=30
Sep 16 19:27:56.082508 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:56.082429    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kube-rbac-proxy" containerID="cri-o://1da890fcca8223ac935f3abec82240b4bb9217befc545e9192c513a067bbd728" gracePeriod=30
Sep 16 19:27:56.421240 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:56.421207    2552 generic.go:356] "Generic (PLEG): container finished" podID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerID="1da890fcca8223ac935f3abec82240b4bb9217befc545e9192c513a067bbd728" exitCode=2
Sep 16 19:27:56.421413 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:56.421278    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" event={"ID":"ce42c78a-330c-47ea-8a8d-587ab85d96b9","Type":"ContainerDied","Data":"1da890fcca8223ac935f3abec82240b4bb9217befc545e9192c513a067bbd728"}
Sep 16 19:27:57.620063 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:57.620028    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:27:57.620539 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:27:57.620229    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:27:58.164752 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:58.164706    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused"
Sep 16 19:27:58.170232 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:58.170200    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 16 19:27:59.141021 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.140983    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:27:59.235041 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.234940    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kube-rbac-proxy-sar-config\") pod \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") "
Sep 16 19:27:59.235041 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.235000    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kserve-provision-location\") pod \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") "
Sep 16 19:27:59.235041 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.235026    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxhbf\" (UniqueName: \"kubernetes.io/projected/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kube-api-access-xxhbf\") pod \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") "
Sep 16 19:27:59.235391 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.235053    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce42c78a-330c-47ea-8a8d-587ab85d96b9-cabundle-cert\") pod \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") "
Sep 16 19:27:59.235391 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.235120    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce42c78a-330c-47ea-8a8d-587ab85d96b9-proxy-tls\") pod \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\" (UID: \"ce42c78a-330c-47ea-8a8d-587ab85d96b9\") "
Sep 16 19:27:59.235509 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.235430    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce42c78a-330c-47ea-8a8d-587ab85d96b9" (UID: "ce42c78a-330c-47ea-8a8d-587ab85d96b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:59.235563 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.235505    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce42c78a-330c-47ea-8a8d-587ab85d96b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ce42c78a-330c-47ea-8a8d-587ab85d96b9" (UID: "ce42c78a-330c-47ea-8a8d-587ab85d96b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:59.237251 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.237226    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce42c78a-330c-47ea-8a8d-587ab85d96b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce42c78a-330c-47ea-8a8d-587ab85d96b9" (UID: "ce42c78a-330c-47ea-8a8d-587ab85d96b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:59.237373 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.237307    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kube-api-access-xxhbf" (OuterVolumeSpecName: "kube-api-access-xxhbf") pod "ce42c78a-330c-47ea-8a8d-587ab85d96b9" (UID: "ce42c78a-330c-47ea-8a8d-587ab85d96b9"). InnerVolumeSpecName "kube-api-access-xxhbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:59.243115 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.243071    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce42c78a-330c-47ea-8a8d-587ab85d96b9" (UID: "ce42c78a-330c-47ea-8a8d-587ab85d96b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:59.336699 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.336657    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:27:59.336699 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.336691    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:27:59.336699 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.336702    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxhbf\" (UniqueName: \"kubernetes.io/projected/ce42c78a-330c-47ea-8a8d-587ab85d96b9-kube-api-access-xxhbf\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:27:59.336699 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.336711    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce42c78a-330c-47ea-8a8d-587ab85d96b9-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:27:59.337021 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.336722    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce42c78a-330c-47ea-8a8d-587ab85d96b9-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:27:59.439334 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.439293    2552 generic.go:356] "Generic (PLEG): container finished" podID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerID="cb7183a74dc56eae30c116ff930a2cd3728aa39b1f3b634b18797e834577b28d" exitCode=0
Sep 16 19:27:59.439565 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.439363    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"
Sep 16 19:27:59.439565 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.439378    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" event={"ID":"ce42c78a-330c-47ea-8a8d-587ab85d96b9","Type":"ContainerDied","Data":"cb7183a74dc56eae30c116ff930a2cd3728aa39b1f3b634b18797e834577b28d"}
Sep 16 19:27:59.439565 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.439448    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6" event={"ID":"ce42c78a-330c-47ea-8a8d-587ab85d96b9","Type":"ContainerDied","Data":"cc82ba4590ba7a99b4f03f854c1565a119eec246ce1ffbf7893e7a2062c1e03e"}
Sep 16 19:27:59.439565 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.439480    2552 scope.go:117] "RemoveContainer" containerID="1da890fcca8223ac935f3abec82240b4bb9217befc545e9192c513a067bbd728"
Sep 16 19:27:59.450246 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.450222    2552 scope.go:117] "RemoveContainer" containerID="cb7183a74dc56eae30c116ff930a2cd3728aa39b1f3b634b18797e834577b28d"
Sep 16 19:27:59.461335 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.461283    2552 scope.go:117] "RemoveContainer" containerID="17007fdb3b990e7c0f6e674adac24dc54a506d54e8d306460cdba044414b1f58"
Sep 16 19:27:59.464552 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.464520    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"]
Sep 16 19:27:59.468675 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.468644    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6"]
Sep 16 19:27:59.471484 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.471470    2552 scope.go:117] "RemoveContainer" containerID="1da890fcca8223ac935f3abec82240b4bb9217befc545e9192c513a067bbd728"
Sep 16 19:27:59.471835 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:27:59.471810    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da890fcca8223ac935f3abec82240b4bb9217befc545e9192c513a067bbd728\": container with ID starting with 1da890fcca8223ac935f3abec82240b4bb9217befc545e9192c513a067bbd728 not found: ID does not exist" containerID="1da890fcca8223ac935f3abec82240b4bb9217befc545e9192c513a067bbd728"
Sep 16 19:27:59.471921 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.471844    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da890fcca8223ac935f3abec82240b4bb9217befc545e9192c513a067bbd728"} err="failed to get container status \"1da890fcca8223ac935f3abec82240b4bb9217befc545e9192c513a067bbd728\": rpc error: code = NotFound desc = could not find container \"1da890fcca8223ac935f3abec82240b4bb9217befc545e9192c513a067bbd728\": container with ID starting with 1da890fcca8223ac935f3abec82240b4bb9217befc545e9192c513a067bbd728 not found: ID does not exist"
Sep 16 19:27:59.471921 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.471864    2552 scope.go:117] "RemoveContainer" containerID="cb7183a74dc56eae30c116ff930a2cd3728aa39b1f3b634b18797e834577b28d"
Sep 16 19:27:59.472199 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:27:59.472175    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7183a74dc56eae30c116ff930a2cd3728aa39b1f3b634b18797e834577b28d\": container with ID starting with cb7183a74dc56eae30c116ff930a2cd3728aa39b1f3b634b18797e834577b28d not found: ID does not exist" containerID="cb7183a74dc56eae30c116ff930a2cd3728aa39b1f3b634b18797e834577b28d"
Sep 16 19:27:59.472254 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.472203    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7183a74dc56eae30c116ff930a2cd3728aa39b1f3b634b18797e834577b28d"} err="failed to get container status \"cb7183a74dc56eae30c116ff930a2cd3728aa39b1f3b634b18797e834577b28d\": rpc error: code = NotFound desc = could not find container \"cb7183a74dc56eae30c116ff930a2cd3728aa39b1f3b634b18797e834577b28d\": container with ID starting with cb7183a74dc56eae30c116ff930a2cd3728aa39b1f3b634b18797e834577b28d not found: ID does not exist"
Sep 16 19:27:59.472254 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.472217    2552 scope.go:117] "RemoveContainer" containerID="17007fdb3b990e7c0f6e674adac24dc54a506d54e8d306460cdba044414b1f58"
Sep 16 19:27:59.472476 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:27:59.472458    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17007fdb3b990e7c0f6e674adac24dc54a506d54e8d306460cdba044414b1f58\": container with ID starting with 17007fdb3b990e7c0f6e674adac24dc54a506d54e8d306460cdba044414b1f58 not found: ID does not exist" containerID="17007fdb3b990e7c0f6e674adac24dc54a506d54e8d306460cdba044414b1f58"
Sep 16 19:27:59.472530 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:27:59.472479    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17007fdb3b990e7c0f6e674adac24dc54a506d54e8d306460cdba044414b1f58"} err="failed to get container status \"17007fdb3b990e7c0f6e674adac24dc54a506d54e8d306460cdba044414b1f58\": rpc error: code = NotFound desc = could not find container \"17007fdb3b990e7c0f6e674adac24dc54a506d54e8d306460cdba044414b1f58\": container with ID starting with 17007fdb3b990e7c0f6e674adac24dc54a506d54e8d306460cdba044414b1f58 not found: ID does not exist"
Sep 16 19:28:00.625096 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:00.625052    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" path="/var/lib/kubelet/pods/ce42c78a-330c-47ea-8a8d-587ab85d96b9/volumes"
Sep 16 19:28:01.256488 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256444    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"]
Sep 16 19:28:01.256789 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256776    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kube-rbac-proxy"
Sep 16 19:28:01.256837 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256791    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kube-rbac-proxy"
Sep 16 19:28:01.256837 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256803    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kserve-container"
Sep 16 19:28:01.256837 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256808    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kserve-container"
Sep 16 19:28:01.256837 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256821    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="storage-initializer"
Sep 16 19:28:01.256837 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256828    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="storage-initializer"
Sep 16 19:28:01.256837 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256835    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kserve-container"
Sep 16 19:28:01.257020 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256841    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kserve-container"
Sep 16 19:28:01.257020 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256853    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kube-rbac-proxy"
Sep 16 19:28:01.257020 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256858    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kube-rbac-proxy"
Sep 16 19:28:01.257020 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256866    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="storage-initializer"
Sep 16 19:28:01.257020 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256871    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="storage-initializer"
Sep 16 19:28:01.257020 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256916    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kube-rbac-proxy"
Sep 16 19:28:01.257020 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256923    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kube-rbac-proxy"
Sep 16 19:28:01.257020 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256929    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e033c16d-de7b-4859-8ee3-6aa647df896c" containerName="kserve-container"
Sep 16 19:28:01.257020 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.256936    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce42c78a-330c-47ea-8a8d-587ab85d96b9" containerName="kserve-container"
Sep 16 19:28:01.260131 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.260082    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.262688 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.262508    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 16 19:28:01.262959 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.262932    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:28:01.263151 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.263127    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:28:01.263267 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.263029    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:28:01.263380 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.263023    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:28:01.263555 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.263532    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:28:01.263826 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.263809    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:28:01.270630 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.270603    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"]
Sep 16 19:28:01.351448 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.351403    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.351633 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.351464    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.351633 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.351549    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn49d\" (UniqueName: \"kubernetes.io/projected/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kube-api-access-pn49d\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.351633 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.351600    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.351633 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.351625    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.452162 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.452121    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.452350 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.452185    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.452350 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.452227    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn49d\" (UniqueName: \"kubernetes.io/projected/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kube-api-access-pn49d\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.452350 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.452273    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.452350 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.452308    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.452577 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:28:01.452458    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 16 19:28:01.452577 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:28:01.452536    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-proxy-tls podName:10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1 nodeName:}" failed. No retries permitted until 2026-09-16 19:28:01.95251419 +0000 UTC m=+2877.846701456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-ffwrv" (UID: "10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 16 19:28:01.452695 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.452632    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.452983 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.452960    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.453034 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.452960    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.463433 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.463404    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn49d\" (UniqueName: \"kubernetes.io/projected/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kube-api-access-pn49d\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.956685 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.956641    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:01.959161 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:01.959136    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ffwrv\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:02.178135 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:02.178054    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:02.452765 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:02.452730    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"]
Sep 16 19:28:02.456595 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:28:02.456569    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ec1f39_ceba_44e4_80cc_b5e4ed15c9a1.slice/crio-5830bd2afd0aa8fc4c2cb6a86f9e8b3a81fc7d2ea0477f3d17be0d3c31dc33f7 WatchSource:0}: Error finding container 5830bd2afd0aa8fc4c2cb6a86f9e8b3a81fc7d2ea0477f3d17be0d3c31dc33f7: Status 404 returned error can't find the container with id 5830bd2afd0aa8fc4c2cb6a86f9e8b3a81fc7d2ea0477f3d17be0d3c31dc33f7
Sep 16 19:28:03.461943 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:03.461905    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" event={"ID":"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1","Type":"ContainerStarted","Data":"e29d61340a2f6c25bdfbc3a17f417c86798e7e68a0eb840326b95bad7f1b68dc"}
Sep 16 19:28:03.462382 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:03.461975    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" event={"ID":"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1","Type":"ContainerStarted","Data":"5830bd2afd0aa8fc4c2cb6a86f9e8b3a81fc7d2ea0477f3d17be0d3c31dc33f7"}
Sep 16 19:28:07.482334 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:07.482240    2552 generic.go:356] "Generic (PLEG): container finished" podID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerID="e29d61340a2f6c25bdfbc3a17f417c86798e7e68a0eb840326b95bad7f1b68dc" exitCode=0
Sep 16 19:28:07.482334 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:07.482300    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" event={"ID":"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1","Type":"ContainerDied","Data":"e29d61340a2f6c25bdfbc3a17f417c86798e7e68a0eb840326b95bad7f1b68dc"}
Sep 16 19:28:08.490696 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:08.490660    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" event={"ID":"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1","Type":"ContainerStarted","Data":"40ebb69917971bbbb1e7153267545bbc9d21208f409775ca3acdbc11aca6878c"}
Sep 16 19:28:08.491111 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:08.490725    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" event={"ID":"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1","Type":"ContainerStarted","Data":"cbadade50e24f5797d8f9615e48f69d75e4b4955ad2e5e957b9d3d8fea168f99"}
Sep 16 19:28:08.491111 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:08.490931    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:08.512399 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:08.512341    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" podStartSLOduration=2.7948145330000003 podStartE2EDuration="7.51232627s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.717511737s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:08.511544687 +0000 UTC m=+2884.405731955" watchObservedRunningTime="2026-09-16 19:28:08.51232627 +0000 UTC m=+2884.406513539"
Sep 16 19:28:09.495547 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:09.495516    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:09.497041 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:09.497011    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 16 19:28:10.499852 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:10.499813    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 16 19:28:12.619948 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:12.619915    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:28:12.620476 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:28:12.620168    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:28:15.505639 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:15.505608    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:15.506331 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:15.506302    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 16 19:28:25.506733 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:25.506696    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 16 19:28:27.619904 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:27.619865    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:28:28.579594 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:28.579552    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerStarted","Data":"341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"}
Sep 16 19:28:28.579792 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:28.579772    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:28:35.506538 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:35.506493    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 16 19:28:45.506788 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:45.506744    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 16 19:28:55.507315 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:55.507275    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:28:59.586195 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:28:59.586158    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:29:02.483536 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:02.483492    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"]
Sep 16 19:29:02.484046 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:02.483789    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kserve-container" containerID="cri-o://cbadade50e24f5797d8f9615e48f69d75e4b4955ad2e5e957b9d3d8fea168f99" gracePeriod=30
Sep 16 19:29:02.484046 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:02.483868    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kube-rbac-proxy" containerID="cri-o://40ebb69917971bbbb1e7153267545bbc9d21208f409775ca3acdbc11aca6878c" gracePeriod=30
Sep 16 19:29:02.735412 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:02.735327    2552 generic.go:356] "Generic (PLEG): container finished" podID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerID="40ebb69917971bbbb1e7153267545bbc9d21208f409775ca3acdbc11aca6878c" exitCode=2
Sep 16 19:29:02.735412 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:02.735401    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" event={"ID":"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1","Type":"ContainerDied","Data":"40ebb69917971bbbb1e7153267545bbc9d21208f409775ca3acdbc11aca6878c"}
Sep 16 19:29:05.500421 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.500376    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Sep 16 19:29:05.506566 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.506539    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 16 19:29:05.635969 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.635938    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:29:05.707452 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.707338    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kserve-provision-location\") pod \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") "
Sep 16 19:29:05.707452 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.707391    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-proxy-tls\") pod \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") "
Sep 16 19:29:05.707452 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.707440    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kube-rbac-proxy-sar-config\") pod \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") "
Sep 16 19:29:05.707771 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.707568    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-cabundle-cert\") pod \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") "
Sep 16 19:29:05.707771 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.707611    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn49d\" (UniqueName: \"kubernetes.io/projected/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kube-api-access-pn49d\") pod \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\" (UID: \"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1\") "
Sep 16 19:29:05.707886 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.707815    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" (UID: "10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:05.707944 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.707880    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" (UID: "10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:05.709647 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.709616    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" (UID: "10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:05.709775 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.709678    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kube-api-access-pn49d" (OuterVolumeSpecName: "kube-api-access-pn49d") pod "10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" (UID: "10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1"). InnerVolumeSpecName "kube-api-access-pn49d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:05.716302 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.716260    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" (UID: "10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:05.751937 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.751903    2552 generic.go:356] "Generic (PLEG): container finished" podID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerID="cbadade50e24f5797d8f9615e48f69d75e4b4955ad2e5e957b9d3d8fea168f99" exitCode=0
Sep 16 19:29:05.752137 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.751962    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"
Sep 16 19:29:05.752137 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.751969    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" event={"ID":"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1","Type":"ContainerDied","Data":"cbadade50e24f5797d8f9615e48f69d75e4b4955ad2e5e957b9d3d8fea168f99"}
Sep 16 19:29:05.752137 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.752018    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv" event={"ID":"10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1","Type":"ContainerDied","Data":"5830bd2afd0aa8fc4c2cb6a86f9e8b3a81fc7d2ea0477f3d17be0d3c31dc33f7"}
Sep 16 19:29:05.752137 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.752044    2552 scope.go:117] "RemoveContainer" containerID="40ebb69917971bbbb1e7153267545bbc9d21208f409775ca3acdbc11aca6878c"
Sep 16 19:29:05.761001 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.760979    2552 scope.go:117] "RemoveContainer" containerID="cbadade50e24f5797d8f9615e48f69d75e4b4955ad2e5e957b9d3d8fea168f99"
Sep 16 19:29:05.769665 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.769646    2552 scope.go:117] "RemoveContainer" containerID="e29d61340a2f6c25bdfbc3a17f417c86798e7e68a0eb840326b95bad7f1b68dc"
Sep 16 19:29:05.777710 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.777689    2552 scope.go:117] "RemoveContainer" containerID="40ebb69917971bbbb1e7153267545bbc9d21208f409775ca3acdbc11aca6878c"
Sep 16 19:29:05.778037 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:29:05.778013    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ebb69917971bbbb1e7153267545bbc9d21208f409775ca3acdbc11aca6878c\": container with ID starting with 40ebb69917971bbbb1e7153267545bbc9d21208f409775ca3acdbc11aca6878c not found: ID does not exist" containerID="40ebb69917971bbbb1e7153267545bbc9d21208f409775ca3acdbc11aca6878c"
Sep 16 19:29:05.778140 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.778050    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ebb69917971bbbb1e7153267545bbc9d21208f409775ca3acdbc11aca6878c"} err="failed to get container status \"40ebb69917971bbbb1e7153267545bbc9d21208f409775ca3acdbc11aca6878c\": rpc error: code = NotFound desc = could not find container \"40ebb69917971bbbb1e7153267545bbc9d21208f409775ca3acdbc11aca6878c\": container with ID starting with 40ebb69917971bbbb1e7153267545bbc9d21208f409775ca3acdbc11aca6878c not found: ID does not exist"
Sep 16 19:29:05.778140 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.778077    2552 scope.go:117] "RemoveContainer" containerID="cbadade50e24f5797d8f9615e48f69d75e4b4955ad2e5e957b9d3d8fea168f99"
Sep 16 19:29:05.778407 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:29:05.778388    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbadade50e24f5797d8f9615e48f69d75e4b4955ad2e5e957b9d3d8fea168f99\": container with ID starting with cbadade50e24f5797d8f9615e48f69d75e4b4955ad2e5e957b9d3d8fea168f99 not found: ID does not exist" containerID="cbadade50e24f5797d8f9615e48f69d75e4b4955ad2e5e957b9d3d8fea168f99"
Sep 16 19:29:05.778507 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.778417    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbadade50e24f5797d8f9615e48f69d75e4b4955ad2e5e957b9d3d8fea168f99"} err="failed to get container status \"cbadade50e24f5797d8f9615e48f69d75e4b4955ad2e5e957b9d3d8fea168f99\": rpc error: code = NotFound desc = could not find container \"cbadade50e24f5797d8f9615e48f69d75e4b4955ad2e5e957b9d3d8fea168f99\": container with ID starting with cbadade50e24f5797d8f9615e48f69d75e4b4955ad2e5e957b9d3d8fea168f99 not found: ID does not exist"
Sep 16 19:29:05.778507 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.778433    2552 scope.go:117] "RemoveContainer" containerID="e29d61340a2f6c25bdfbc3a17f417c86798e7e68a0eb840326b95bad7f1b68dc"
Sep 16 19:29:05.778718 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:29:05.778700    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e29d61340a2f6c25bdfbc3a17f417c86798e7e68a0eb840326b95bad7f1b68dc\": container with ID starting with e29d61340a2f6c25bdfbc3a17f417c86798e7e68a0eb840326b95bad7f1b68dc not found: ID does not exist" containerID="e29d61340a2f6c25bdfbc3a17f417c86798e7e68a0eb840326b95bad7f1b68dc"
Sep 16 19:29:05.778767 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.778723    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e29d61340a2f6c25bdfbc3a17f417c86798e7e68a0eb840326b95bad7f1b68dc"} err="failed to get container status \"e29d61340a2f6c25bdfbc3a17f417c86798e7e68a0eb840326b95bad7f1b68dc\": rpc error: code = NotFound desc = could not find container \"e29d61340a2f6c25bdfbc3a17f417c86798e7e68a0eb840326b95bad7f1b68dc\": container with ID starting with e29d61340a2f6c25bdfbc3a17f417c86798e7e68a0eb840326b95bad7f1b68dc not found: ID does not exist"
Sep 16 19:29:05.786865 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.786834    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"]
Sep 16 19:29:05.795252 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.795219    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv"]
Sep 16 19:29:05.808840 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.808805    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:29:05.808840 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.808836    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn49d\" (UniqueName: \"kubernetes.io/projected/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kube-api-access-pn49d\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:29:05.808840 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.808846    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:29:05.809046 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.808858    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:29:05.809046 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:05.808869    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:29:06.625205 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:06.625168    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" path="/var/lib/kubelet/pods/10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1/volumes"
Sep 16 19:29:07.657731 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.657693    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"]
Sep 16 19:29:07.658161 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.658000    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="storage-initializer"
Sep 16 19:29:07.658161 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.658011    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="storage-initializer"
Sep 16 19:29:07.658161 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.658023    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kube-rbac-proxy"
Sep 16 19:29:07.658161 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.658028    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kube-rbac-proxy"
Sep 16 19:29:07.658161 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.658039    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kserve-container"
Sep 16 19:29:07.658161 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.658045    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kserve-container"
Sep 16 19:29:07.658161 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.658118    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kube-rbac-proxy"
Sep 16 19:29:07.658161 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.658127    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="10ec1f39-ceba-44e4-80cc-b5e4ed15c9a1" containerName="kserve-container"
Sep 16 19:29:07.661209 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.661192    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:07.663376 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.663345    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 16 19:29:07.663524 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.663385    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:29:07.663524 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.663437    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:29:07.663644 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.663602    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:29:07.664251 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.664232    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:29:07.664352 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.664265    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:29:07.664352 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.664270    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 19:29:07.672109 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.672064    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"]
Sep 16 19:29:07.725023 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.724991    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2650f1e8-85fe-4412-b43b-abe4680c6549-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:07.725165 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.725038    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2650f1e8-85fe-4412-b43b-abe4680c6549-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:07.725165 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.725062    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9s6v\" (UniqueName: \"kubernetes.io/projected/2650f1e8-85fe-4412-b43b-abe4680c6549-kube-api-access-w9s6v\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:07.725165 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.725101    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2650f1e8-85fe-4412-b43b-abe4680c6549-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:07.725165 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.725122    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2650f1e8-85fe-4412-b43b-abe4680c6549-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:07.825588 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.825538    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2650f1e8-85fe-4412-b43b-abe4680c6549-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:07.825588 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.825593    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2650f1e8-85fe-4412-b43b-abe4680c6549-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:07.825786 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.825719    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9s6v\" (UniqueName: \"kubernetes.io/projected/2650f1e8-85fe-4412-b43b-abe4680c6549-kube-api-access-w9s6v\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:07.825786 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.825778    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2650f1e8-85fe-4412-b43b-abe4680c6549-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:07.825876 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.825815    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2650f1e8-85fe-4412-b43b-abe4680c6549-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:07.825915 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:29:07.825893    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 16 19:29:07.825965 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:29:07.825953    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2650f1e8-85fe-4412-b43b-abe4680c6549-proxy-tls podName:2650f1e8-85fe-4412-b43b-abe4680c6549 nodeName:}" failed. No retries permitted until 2026-09-16 19:29:08.325936482 +0000 UTC m=+2944.220123733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2650f1e8-85fe-4412-b43b-abe4680c6549-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" (UID: "2650f1e8-85fe-4412-b43b-abe4680c6549") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 16 19:29:07.826264 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.826242    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2650f1e8-85fe-4412-b43b-abe4680c6549-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:07.826425 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.826409    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2650f1e8-85fe-4412-b43b-abe4680c6549-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:07.826465 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.826410    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2650f1e8-85fe-4412-b43b-abe4680c6549-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:07.834889 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:07.834857    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9s6v\" (UniqueName: \"kubernetes.io/projected/2650f1e8-85fe-4412-b43b-abe4680c6549-kube-api-access-w9s6v\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:08.330869 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:08.330826    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2650f1e8-85fe-4412-b43b-abe4680c6549-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:08.333282 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:08.333257    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2650f1e8-85fe-4412-b43b-abe4680c6549-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:08.572258 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:08.572219    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:08.849230 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:08.849197    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"]
Sep 16 19:29:08.851740 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:29:08.851712    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2650f1e8_85fe_4412_b43b_abe4680c6549.slice/crio-09df1867796d44d3506e84eaf9a15e3f7a116f504cc224764eda3691b3d9ad86 WatchSource:0}: Error finding container 09df1867796d44d3506e84eaf9a15e3f7a116f504cc224764eda3691b3d9ad86: Status 404 returned error can't find the container with id 09df1867796d44d3506e84eaf9a15e3f7a116f504cc224764eda3691b3d9ad86
Sep 16 19:29:09.774438 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:09.774400    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" event={"ID":"2650f1e8-85fe-4412-b43b-abe4680c6549","Type":"ContainerStarted","Data":"6b1bb63e5459b575968b82b8c73f7da391ef2802c52aef4f2fe9bb15e90b9e45"}
Sep 16 19:29:09.774638 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:09.774462    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" event={"ID":"2650f1e8-85fe-4412-b43b-abe4680c6549","Type":"ContainerStarted","Data":"09df1867796d44d3506e84eaf9a15e3f7a116f504cc224764eda3691b3d9ad86"}
Sep 16 19:29:13.795456 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:13.795425    2552 generic.go:356] "Generic (PLEG): container finished" podID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerID="6b1bb63e5459b575968b82b8c73f7da391ef2802c52aef4f2fe9bb15e90b9e45" exitCode=0
Sep 16 19:29:13.795838 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:13.795502    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" event={"ID":"2650f1e8-85fe-4412-b43b-abe4680c6549","Type":"ContainerDied","Data":"6b1bb63e5459b575968b82b8c73f7da391ef2802c52aef4f2fe9bb15e90b9e45"}
Sep 16 19:29:14.803081 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:14.803048    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" event={"ID":"2650f1e8-85fe-4412-b43b-abe4680c6549","Type":"ContainerStarted","Data":"98f9246463fe9e29b986e96541bfe819f0d926cb12f326b78a97ee957dafdb24"}
Sep 16 19:29:14.803477 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:14.803138    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" event={"ID":"2650f1e8-85fe-4412-b43b-abe4680c6549","Type":"ContainerStarted","Data":"c42dd4494aaaa06cb1399ff76cb1868e8f999143415daed16b8827a355843d0d"}
Sep 16 19:29:14.803477 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:14.803370    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:14.827587 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:14.827526    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" podStartSLOduration=3.4316885839999998 podStartE2EDuration="7.827507623s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.395819039s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:14.826062475 +0000 UTC m=+2950.720249737" watchObservedRunningTime="2026-09-16 19:29:14.827507623 +0000 UTC m=+2950.721694891"
Sep 16 19:29:15.807478 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:15.807439    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:15.808981 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:15.808961    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 16 19:29:16.811011 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:16.810969    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 16 19:29:21.816371 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:21.816342    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:29:21.816842 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:21.816814    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 16 19:29:31.817757 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:31.817676    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 16 19:29:41.816863 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:41.816821    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 16 19:29:51.817356 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:29:51.817317    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 16 19:30:01.818199 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:01.818164    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:30:08.993556 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:08.993512    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"]
Sep 16 19:30:08.994079 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:08.993789    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kserve-container" containerID="cri-o://c42dd4494aaaa06cb1399ff76cb1868e8f999143415daed16b8827a355843d0d" gracePeriod=30
Sep 16 19:30:08.994079 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:08.993830    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kube-rbac-proxy" containerID="cri-o://98f9246463fe9e29b986e96541bfe819f0d926cb12f326b78a97ee957dafdb24" gracePeriod=30
Sep 16 19:30:10.057752 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:10.057713    2552 generic.go:356] "Generic (PLEG): container finished" podID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerID="98f9246463fe9e29b986e96541bfe819f0d926cb12f326b78a97ee957dafdb24" exitCode=2
Sep 16 19:30:10.058150 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:10.057765    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" event={"ID":"2650f1e8-85fe-4412-b43b-abe4680c6549","Type":"ContainerDied","Data":"98f9246463fe9e29b986e96541bfe819f0d926cb12f326b78a97ee957dafdb24"}
Sep 16 19:30:11.811803 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:11.811756    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.63:8643/healthz\": dial tcp 10.132.0.63:8643: connect: connection refused"
Sep 16 19:30:11.817152 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:11.817126    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 16 19:30:12.038443 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.038420    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:30:12.068199 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.068130    2552 generic.go:356] "Generic (PLEG): container finished" podID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerID="c42dd4494aaaa06cb1399ff76cb1868e8f999143415daed16b8827a355843d0d" exitCode=0
Sep 16 19:30:12.068199 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.068167    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" event={"ID":"2650f1e8-85fe-4412-b43b-abe4680c6549","Type":"ContainerDied","Data":"c42dd4494aaaa06cb1399ff76cb1868e8f999143415daed16b8827a355843d0d"}
Sep 16 19:30:12.068348 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.068200    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"
Sep 16 19:30:12.068348 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.068220    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv" event={"ID":"2650f1e8-85fe-4412-b43b-abe4680c6549","Type":"ContainerDied","Data":"09df1867796d44d3506e84eaf9a15e3f7a116f504cc224764eda3691b3d9ad86"}
Sep 16 19:30:12.068348 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.068248    2552 scope.go:117] "RemoveContainer" containerID="98f9246463fe9e29b986e96541bfe819f0d926cb12f326b78a97ee957dafdb24"
Sep 16 19:30:12.076986 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.076958    2552 scope.go:117] "RemoveContainer" containerID="c42dd4494aaaa06cb1399ff76cb1868e8f999143415daed16b8827a355843d0d"
Sep 16 19:30:12.077969 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.077943    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9s6v\" (UniqueName: \"kubernetes.io/projected/2650f1e8-85fe-4412-b43b-abe4680c6549-kube-api-access-w9s6v\") pod \"2650f1e8-85fe-4412-b43b-abe4680c6549\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") "
Sep 16 19:30:12.078034 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.078012    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2650f1e8-85fe-4412-b43b-abe4680c6549-kserve-provision-location\") pod \"2650f1e8-85fe-4412-b43b-abe4680c6549\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") "
Sep 16 19:30:12.078113 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.078050    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2650f1e8-85fe-4412-b43b-abe4680c6549-cabundle-cert\") pod \"2650f1e8-85fe-4412-b43b-abe4680c6549\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") "
Sep 16 19:30:12.078184 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.078132    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2650f1e8-85fe-4412-b43b-abe4680c6549-proxy-tls\") pod \"2650f1e8-85fe-4412-b43b-abe4680c6549\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") "
Sep 16 19:30:12.078243 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.078192    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2650f1e8-85fe-4412-b43b-abe4680c6549-kube-rbac-proxy-sar-config\") pod \"2650f1e8-85fe-4412-b43b-abe4680c6549\" (UID: \"2650f1e8-85fe-4412-b43b-abe4680c6549\") "
Sep 16 19:30:12.078607 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.078534    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2650f1e8-85fe-4412-b43b-abe4680c6549-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2650f1e8-85fe-4412-b43b-abe4680c6549" (UID: "2650f1e8-85fe-4412-b43b-abe4680c6549"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:12.078704 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.078609    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2650f1e8-85fe-4412-b43b-abe4680c6549-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2650f1e8-85fe-4412-b43b-abe4680c6549" (UID: "2650f1e8-85fe-4412-b43b-abe4680c6549"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:12.080445 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.080409    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2650f1e8-85fe-4412-b43b-abe4680c6549-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2650f1e8-85fe-4412-b43b-abe4680c6549" (UID: "2650f1e8-85fe-4412-b43b-abe4680c6549"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:12.080754 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.080736    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2650f1e8-85fe-4412-b43b-abe4680c6549-kube-api-access-w9s6v" (OuterVolumeSpecName: "kube-api-access-w9s6v") pod "2650f1e8-85fe-4412-b43b-abe4680c6549" (UID: "2650f1e8-85fe-4412-b43b-abe4680c6549"). InnerVolumeSpecName "kube-api-access-w9s6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:12.088215 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.088192    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2650f1e8-85fe-4412-b43b-abe4680c6549-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2650f1e8-85fe-4412-b43b-abe4680c6549" (UID: "2650f1e8-85fe-4412-b43b-abe4680c6549"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:12.094502 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.094482    2552 scope.go:117] "RemoveContainer" containerID="6b1bb63e5459b575968b82b8c73f7da391ef2802c52aef4f2fe9bb15e90b9e45"
Sep 16 19:30:12.101965 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.101949    2552 scope.go:117] "RemoveContainer" containerID="98f9246463fe9e29b986e96541bfe819f0d926cb12f326b78a97ee957dafdb24"
Sep 16 19:30:12.102246 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:30:12.102226    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f9246463fe9e29b986e96541bfe819f0d926cb12f326b78a97ee957dafdb24\": container with ID starting with 98f9246463fe9e29b986e96541bfe819f0d926cb12f326b78a97ee957dafdb24 not found: ID does not exist" containerID="98f9246463fe9e29b986e96541bfe819f0d926cb12f326b78a97ee957dafdb24"
Sep 16 19:30:12.102301 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.102253    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f9246463fe9e29b986e96541bfe819f0d926cb12f326b78a97ee957dafdb24"} err="failed to get container status \"98f9246463fe9e29b986e96541bfe819f0d926cb12f326b78a97ee957dafdb24\": rpc error: code = NotFound desc = could not find container \"98f9246463fe9e29b986e96541bfe819f0d926cb12f326b78a97ee957dafdb24\": container with ID starting with 98f9246463fe9e29b986e96541bfe819f0d926cb12f326b78a97ee957dafdb24 not found: ID does not exist"
Sep 16 19:30:12.102301 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.102271    2552 scope.go:117] "RemoveContainer" containerID="c42dd4494aaaa06cb1399ff76cb1868e8f999143415daed16b8827a355843d0d"
Sep 16 19:30:12.102533 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:30:12.102515    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42dd4494aaaa06cb1399ff76cb1868e8f999143415daed16b8827a355843d0d\": container with ID starting with c42dd4494aaaa06cb1399ff76cb1868e8f999143415daed16b8827a355843d0d not found: ID does not exist" containerID="c42dd4494aaaa06cb1399ff76cb1868e8f999143415daed16b8827a355843d0d"
Sep 16 19:30:12.102582 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.102538    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42dd4494aaaa06cb1399ff76cb1868e8f999143415daed16b8827a355843d0d"} err="failed to get container status \"c42dd4494aaaa06cb1399ff76cb1868e8f999143415daed16b8827a355843d0d\": rpc error: code = NotFound desc = could not find container \"c42dd4494aaaa06cb1399ff76cb1868e8f999143415daed16b8827a355843d0d\": container with ID starting with c42dd4494aaaa06cb1399ff76cb1868e8f999143415daed16b8827a355843d0d not found: ID does not exist"
Sep 16 19:30:12.102582 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.102554    2552 scope.go:117] "RemoveContainer" containerID="6b1bb63e5459b575968b82b8c73f7da391ef2802c52aef4f2fe9bb15e90b9e45"
Sep 16 19:30:12.102808 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:30:12.102790    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b1bb63e5459b575968b82b8c73f7da391ef2802c52aef4f2fe9bb15e90b9e45\": container with ID starting with 6b1bb63e5459b575968b82b8c73f7da391ef2802c52aef4f2fe9bb15e90b9e45 not found: ID does not exist" containerID="6b1bb63e5459b575968b82b8c73f7da391ef2802c52aef4f2fe9bb15e90b9e45"
Sep 16 19:30:12.102851 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.102813    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1bb63e5459b575968b82b8c73f7da391ef2802c52aef4f2fe9bb15e90b9e45"} err="failed to get container status \"6b1bb63e5459b575968b82b8c73f7da391ef2802c52aef4f2fe9bb15e90b9e45\": rpc error: code = NotFound desc = could not find container \"6b1bb63e5459b575968b82b8c73f7da391ef2802c52aef4f2fe9bb15e90b9e45\": container with ID starting with 6b1bb63e5459b575968b82b8c73f7da391ef2802c52aef4f2fe9bb15e90b9e45 not found: ID does not exist"
Sep 16 19:30:12.179108 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.179055    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2650f1e8-85fe-4412-b43b-abe4680c6549-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:30:12.179108 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.179078    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2650f1e8-85fe-4412-b43b-abe4680c6549-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:30:12.179108 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.179099    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2650f1e8-85fe-4412-b43b-abe4680c6549-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:30:12.179108 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.179107    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2650f1e8-85fe-4412-b43b-abe4680c6549-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:30:12.179108 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.179115    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9s6v\" (UniqueName: \"kubernetes.io/projected/2650f1e8-85fe-4412-b43b-abe4680c6549-kube-api-access-w9s6v\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:30:12.392449 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.392413    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"]
Sep 16 19:30:12.397397 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.397366    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv"]
Sep 16 19:30:12.625989 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:12.625908    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" path="/var/lib/kubelet/pods/2650f1e8-85fe-4412-b43b-abe4680c6549/volumes"
Sep 16 19:30:14.268369 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.268331    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"]
Sep 16 19:30:14.268750 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.268664    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kserve-container"
Sep 16 19:30:14.268750 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.268676    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kserve-container"
Sep 16 19:30:14.268750 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.268685    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="storage-initializer"
Sep 16 19:30:14.268750 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.268692    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="storage-initializer"
Sep 16 19:30:14.268750 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.268711    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kube-rbac-proxy"
Sep 16 19:30:14.268750 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.268717    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kube-rbac-proxy"
Sep 16 19:30:14.268952 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.268774    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kserve-container"
Sep 16 19:30:14.268952 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.268783    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2650f1e8-85fe-4412-b43b-abe4680c6549" containerName="kube-rbac-proxy"
Sep 16 19:30:14.272420 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.272403    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.275062 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.275040    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 16 19:30:14.275191 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.275118    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:30:14.275191 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.275175    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 16 19:30:14.275372 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.275353    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:30:14.275536 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.275516    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:30:14.276003 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.275989    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:30:14.276051 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.275992    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:30:14.284443 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.284422    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"]
Sep 16 19:30:14.397923 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.397882    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64996fab-1c6f-4ae5-aa04-632ddd689846-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-6rsz5\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.398151 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.397943    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64996fab-1c6f-4ae5-aa04-632ddd689846-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-6rsz5\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.398151 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.397992    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64996fab-1c6f-4ae5-aa04-632ddd689846-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-6rsz5\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.398151 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.398015    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64996fab-1c6f-4ae5-aa04-632ddd689846-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-6rsz5\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.398151 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.398071    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh8sz\" (UniqueName: \"kubernetes.io/projected/64996fab-1c6f-4ae5-aa04-632ddd689846-kube-api-access-nh8sz\") pod \"isvc-pmml-predictor-8674795f59-6rsz5\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.499387 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.499355    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh8sz\" (UniqueName: \"kubernetes.io/projected/64996fab-1c6f-4ae5-aa04-632ddd689846-kube-api-access-nh8sz\") pod \"isvc-pmml-predictor-8674795f59-6rsz5\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.499566 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.499398    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64996fab-1c6f-4ae5-aa04-632ddd689846-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-6rsz5\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.499566 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.499444    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64996fab-1c6f-4ae5-aa04-632ddd689846-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-6rsz5\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.499566 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.499465    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64996fab-1c6f-4ae5-aa04-632ddd689846-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-6rsz5\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.499566 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.499480    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64996fab-1c6f-4ae5-aa04-632ddd689846-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-6rsz5\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.499908 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.499882    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64996fab-1c6f-4ae5-aa04-632ddd689846-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-6rsz5\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.500170 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.500149    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64996fab-1c6f-4ae5-aa04-632ddd689846-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-6rsz5\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.500225 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.500176    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64996fab-1c6f-4ae5-aa04-632ddd689846-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-6rsz5\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.502001 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.501977    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64996fab-1c6f-4ae5-aa04-632ddd689846-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-6rsz5\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.510068 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.510041    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh8sz\" (UniqueName: \"kubernetes.io/projected/64996fab-1c6f-4ae5-aa04-632ddd689846-kube-api-access-nh8sz\") pod \"isvc-pmml-predictor-8674795f59-6rsz5\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.582329 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.582238    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:14.860784 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:14.860755    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"]
Sep 16 19:30:14.863530 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:30:14.863490    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64996fab_1c6f_4ae5_aa04_632ddd689846.slice/crio-21342cf5716bf46364849d207fba9e5bfaf1d43b79d988c28ccddb7278b58285 WatchSource:0}: Error finding container 21342cf5716bf46364849d207fba9e5bfaf1d43b79d988c28ccddb7278b58285: Status 404 returned error can't find the container with id 21342cf5716bf46364849d207fba9e5bfaf1d43b79d988c28ccddb7278b58285
Sep 16 19:30:15.086897 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:15.086854    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" event={"ID":"64996fab-1c6f-4ae5-aa04-632ddd689846","Type":"ContainerStarted","Data":"9118325ce28d71d8c7130b617202c8e82093d9e361a0c5200f91607e7b3fa4d9"}
Sep 16 19:30:15.087137 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:15.086921    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" event={"ID":"64996fab-1c6f-4ae5-aa04-632ddd689846","Type":"ContainerStarted","Data":"21342cf5716bf46364849d207fba9e5bfaf1d43b79d988c28ccddb7278b58285"}
Sep 16 19:30:19.106118 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:19.106014    2552 generic.go:356] "Generic (PLEG): container finished" podID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerID="9118325ce28d71d8c7130b617202c8e82093d9e361a0c5200f91607e7b3fa4d9" exitCode=0
Sep 16 19:30:19.106472 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:19.106113    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" event={"ID":"64996fab-1c6f-4ae5-aa04-632ddd689846","Type":"ContainerDied","Data":"9118325ce28d71d8c7130b617202c8e82093d9e361a0c5200f91607e7b3fa4d9"}
Sep 16 19:30:27.152260 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:27.152223    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" event={"ID":"64996fab-1c6f-4ae5-aa04-632ddd689846","Type":"ContainerStarted","Data":"4bb450b8faaa6941a32ef065e0e9e180dab58e0c9669fbde6fb69949158d031f"}
Sep 16 19:30:27.152655 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:27.152287    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" event={"ID":"64996fab-1c6f-4ae5-aa04-632ddd689846","Type":"ContainerStarted","Data":"b4df2e0d77a4e554bf19f8a5b55dc36cbdf0173e6ad2d821027a01a720c4ec02"}
Sep 16 19:30:27.152655 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:27.152480    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:27.173360 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:27.173296    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" podStartSLOduration=2.164183313 podStartE2EDuration="13.173279377s" totalImagesPullingTime="7.17723412s" totalInitContainerRuntime="3.831861944s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:30:27.172044323 +0000 UTC m=+3023.066231592" watchObservedRunningTime="2026-09-16 19:30:27.173279377 +0000 UTC m=+3023.067466647"
Sep 16 19:30:28.156630 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:28.156600    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:28.157994 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:28.157967    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 19:30:29.160919 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:29.160877    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 19:30:32.928383 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:32.928347    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"]
Sep 16 19:30:32.931638 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:32.931613    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:32.933922 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:32.933895    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:30:32.934057 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:32.933895    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 16 19:30:32.934057 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:32.933956    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 16 19:30:32.944477 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:32.944454    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"]
Sep 16 19:30:33.067407 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:33.067368    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6df1e4bf-f13b-48ad-955c-c145483569a3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:33.067580 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:33.067415    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df1e4bf-f13b-48ad-955c-c145483569a3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:33.067580 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:33.067446    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqjh6\" (UniqueName: \"kubernetes.io/projected/6df1e4bf-f13b-48ad-955c-c145483569a3-kube-api-access-gqjh6\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:33.067580 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:33.067525    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6df1e4bf-f13b-48ad-955c-c145483569a3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:33.168807 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:33.168766    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqjh6\" (UniqueName: \"kubernetes.io/projected/6df1e4bf-f13b-48ad-955c-c145483569a3-kube-api-access-gqjh6\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:33.169032 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:33.168837    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6df1e4bf-f13b-48ad-955c-c145483569a3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:33.169032 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:33.168866    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6df1e4bf-f13b-48ad-955c-c145483569a3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:33.169032 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:33.168896    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df1e4bf-f13b-48ad-955c-c145483569a3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:33.169032 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:30:33.168990    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 16 19:30:33.169356 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:30:33.169059    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df1e4bf-f13b-48ad-955c-c145483569a3-proxy-tls podName:6df1e4bf-f13b-48ad-955c-c145483569a3 nodeName:}" failed. No retries permitted until 2026-09-16 19:30:33.669039678 +0000 UTC m=+3029.563226924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6df1e4bf-f13b-48ad-955c-c145483569a3-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch" (UID: "6df1e4bf-f13b-48ad-955c-c145483569a3") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 16 19:30:33.169433 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:33.169405    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6df1e4bf-f13b-48ad-955c-c145483569a3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:33.169607 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:33.169585    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6df1e4bf-f13b-48ad-955c-c145483569a3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:33.180154 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:33.180078    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqjh6\" (UniqueName: \"kubernetes.io/projected/6df1e4bf-f13b-48ad-955c-c145483569a3-kube-api-access-gqjh6\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:33.672862 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:33.672816    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df1e4bf-f13b-48ad-955c-c145483569a3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:33.675414 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:33.675389    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df1e4bf-f13b-48ad-955c-c145483569a3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:33.842211 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:33.842165    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:34.127768 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:34.127682    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"]
Sep 16 19:30:34.132574 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:30:34.132540    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df1e4bf_f13b_48ad_955c_c145483569a3.slice/crio-4ee629d2c5146e7dafe091b129e42133465cf4d9e19203a0fa91f658c33e9c91 WatchSource:0}: Error finding container 4ee629d2c5146e7dafe091b129e42133465cf4d9e19203a0fa91f658c33e9c91: Status 404 returned error can't find the container with id 4ee629d2c5146e7dafe091b129e42133465cf4d9e19203a0fa91f658c33e9c91
Sep 16 19:30:34.167250 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:34.167222    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:30:34.167746 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:34.167720    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 19:30:34.187045 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:34.187003    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch" event={"ID":"6df1e4bf-f13b-48ad-955c-c145483569a3","Type":"ContainerStarted","Data":"4ee629d2c5146e7dafe091b129e42133465cf4d9e19203a0fa91f658c33e9c91"}
Sep 16 19:30:35.193678 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:35.193633    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch" event={"ID":"6df1e4bf-f13b-48ad-955c-c145483569a3","Type":"ContainerStarted","Data":"3e0588c229328db54af5273618adf53789a5bc1646116037fa78337225da59fd"}
Sep 16 19:30:39.215423 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:39.215388    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch_6df1e4bf-f13b-48ad-955c-c145483569a3/storage-initializer/0.log"
Sep 16 19:30:39.215849 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:39.215437    2552 generic.go:356] "Generic (PLEG): container finished" podID="6df1e4bf-f13b-48ad-955c-c145483569a3" containerID="3e0588c229328db54af5273618adf53789a5bc1646116037fa78337225da59fd" exitCode=1
Sep 16 19:30:39.215849 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:39.215481    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch" event={"ID":"6df1e4bf-f13b-48ad-955c-c145483569a3","Type":"ContainerDied","Data":"3e0588c229328db54af5273618adf53789a5bc1646116037fa78337225da59fd"}
Sep 16 19:30:40.221968 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:40.221940    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch_6df1e4bf-f13b-48ad-955c-c145483569a3/storage-initializer/0.log"
Sep 16 19:30:40.222407 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:40.222039    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch" event={"ID":"6df1e4bf-f13b-48ad-955c-c145483569a3","Type":"ContainerStarted","Data":"0b780871688b3913c23b327f1d76f72cbb9bd405c77edeca57312ebc33928283"}
Sep 16 19:30:42.888839 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:42.888804    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"]
Sep 16 19:30:42.889262 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:42.889027    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch" podUID="6df1e4bf-f13b-48ad-955c-c145483569a3" containerName="storage-initializer" containerID="cri-o://0b780871688b3913c23b327f1d76f72cbb9bd405c77edeca57312ebc33928283" gracePeriod=30
Sep 16 19:30:43.025895 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.025871    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch_6df1e4bf-f13b-48ad-955c-c145483569a3/storage-initializer/1.log"
Sep 16 19:30:43.026311 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.026296    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch_6df1e4bf-f13b-48ad-955c-c145483569a3/storage-initializer/0.log"
Sep 16 19:30:43.028696 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.028680    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:43.149788 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.149681    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqjh6\" (UniqueName: \"kubernetes.io/projected/6df1e4bf-f13b-48ad-955c-c145483569a3-kube-api-access-gqjh6\") pod \"6df1e4bf-f13b-48ad-955c-c145483569a3\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") "
Sep 16 19:30:43.149788 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.149740    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6df1e4bf-f13b-48ad-955c-c145483569a3-kube-rbac-proxy-sar-config\") pod \"6df1e4bf-f13b-48ad-955c-c145483569a3\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") "
Sep 16 19:30:43.149788 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.149775    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df1e4bf-f13b-48ad-955c-c145483569a3-proxy-tls\") pod \"6df1e4bf-f13b-48ad-955c-c145483569a3\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") "
Sep 16 19:30:43.150075 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.149892    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6df1e4bf-f13b-48ad-955c-c145483569a3-kserve-provision-location\") pod \"6df1e4bf-f13b-48ad-955c-c145483569a3\" (UID: \"6df1e4bf-f13b-48ad-955c-c145483569a3\") "
Sep 16 19:30:43.150237 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.150210    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df1e4bf-f13b-48ad-955c-c145483569a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6df1e4bf-f13b-48ad-955c-c145483569a3" (UID: "6df1e4bf-f13b-48ad-955c-c145483569a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:43.150292 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.150230    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df1e4bf-f13b-48ad-955c-c145483569a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6df1e4bf-f13b-48ad-955c-c145483569a3" (UID: "6df1e4bf-f13b-48ad-955c-c145483569a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:43.151859 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.151832    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df1e4bf-f13b-48ad-955c-c145483569a3-kube-api-access-gqjh6" (OuterVolumeSpecName: "kube-api-access-gqjh6") pod "6df1e4bf-f13b-48ad-955c-c145483569a3" (UID: "6df1e4bf-f13b-48ad-955c-c145483569a3"). InnerVolumeSpecName "kube-api-access-gqjh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:43.151971 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.151930    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df1e4bf-f13b-48ad-955c-c145483569a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6df1e4bf-f13b-48ad-955c-c145483569a3" (UID: "6df1e4bf-f13b-48ad-955c-c145483569a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:43.237956 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.237929    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch_6df1e4bf-f13b-48ad-955c-c145483569a3/storage-initializer/1.log"
Sep 16 19:30:43.238470 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.238450    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch_6df1e4bf-f13b-48ad-955c-c145483569a3/storage-initializer/0.log"
Sep 16 19:30:43.238537 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.238499    2552 generic.go:356] "Generic (PLEG): container finished" podID="6df1e4bf-f13b-48ad-955c-c145483569a3" containerID="0b780871688b3913c23b327f1d76f72cbb9bd405c77edeca57312ebc33928283" exitCode=1
Sep 16 19:30:43.238575 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.238538    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch" event={"ID":"6df1e4bf-f13b-48ad-955c-c145483569a3","Type":"ContainerDied","Data":"0b780871688b3913c23b327f1d76f72cbb9bd405c77edeca57312ebc33928283"}
Sep 16 19:30:43.238575 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.238560    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"
Sep 16 19:30:43.238645 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.238632    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch" event={"ID":"6df1e4bf-f13b-48ad-955c-c145483569a3","Type":"ContainerDied","Data":"4ee629d2c5146e7dafe091b129e42133465cf4d9e19203a0fa91f658c33e9c91"}
Sep 16 19:30:43.238682 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.238661    2552 scope.go:117] "RemoveContainer" containerID="0b780871688b3913c23b327f1d76f72cbb9bd405c77edeca57312ebc33928283"
Sep 16 19:30:43.248184 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.248158    2552 scope.go:117] "RemoveContainer" containerID="3e0588c229328db54af5273618adf53789a5bc1646116037fa78337225da59fd"
Sep 16 19:30:43.250747 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.250717    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6df1e4bf-f13b-48ad-955c-c145483569a3-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:30:43.250871 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.250751    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqjh6\" (UniqueName: \"kubernetes.io/projected/6df1e4bf-f13b-48ad-955c-c145483569a3-kube-api-access-gqjh6\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:30:43.250871 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.250768    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6df1e4bf-f13b-48ad-955c-c145483569a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:30:43.250871 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.250782    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df1e4bf-f13b-48ad-955c-c145483569a3-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:30:43.256719 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.256693    2552 scope.go:117] "RemoveContainer" containerID="0b780871688b3913c23b327f1d76f72cbb9bd405c77edeca57312ebc33928283"
Sep 16 19:30:43.257062 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:30:43.257042    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b780871688b3913c23b327f1d76f72cbb9bd405c77edeca57312ebc33928283\": container with ID starting with 0b780871688b3913c23b327f1d76f72cbb9bd405c77edeca57312ebc33928283 not found: ID does not exist" containerID="0b780871688b3913c23b327f1d76f72cbb9bd405c77edeca57312ebc33928283"
Sep 16 19:30:43.257178 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.257078    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b780871688b3913c23b327f1d76f72cbb9bd405c77edeca57312ebc33928283"} err="failed to get container status \"0b780871688b3913c23b327f1d76f72cbb9bd405c77edeca57312ebc33928283\": rpc error: code = NotFound desc = could not find container \"0b780871688b3913c23b327f1d76f72cbb9bd405c77edeca57312ebc33928283\": container with ID starting with 0b780871688b3913c23b327f1d76f72cbb9bd405c77edeca57312ebc33928283 not found: ID does not exist"
Sep 16 19:30:43.257178 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.257112    2552 scope.go:117] "RemoveContainer" containerID="3e0588c229328db54af5273618adf53789a5bc1646116037fa78337225da59fd"
Sep 16 19:30:43.257404 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:30:43.257387    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e0588c229328db54af5273618adf53789a5bc1646116037fa78337225da59fd\": container with ID starting with 3e0588c229328db54af5273618adf53789a5bc1646116037fa78337225da59fd not found: ID does not exist" containerID="3e0588c229328db54af5273618adf53789a5bc1646116037fa78337225da59fd"
Sep 16 19:30:43.257448 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.257414    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e0588c229328db54af5273618adf53789a5bc1646116037fa78337225da59fd"} err="failed to get container status \"3e0588c229328db54af5273618adf53789a5bc1646116037fa78337225da59fd\": rpc error: code = NotFound desc = could not find container \"3e0588c229328db54af5273618adf53789a5bc1646116037fa78337225da59fd\": container with ID starting with 3e0588c229328db54af5273618adf53789a5bc1646116037fa78337225da59fd not found: ID does not exist"
Sep 16 19:30:43.279685 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.279649    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"]
Sep 16 19:30:43.284530 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:43.284496    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch"]
Sep 16 19:30:44.047854 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.047815    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"]
Sep 16 19:30:44.048262 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.048165    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df1e4bf-f13b-48ad-955c-c145483569a3" containerName="storage-initializer"
Sep 16 19:30:44.048262 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.048178    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df1e4bf-f13b-48ad-955c-c145483569a3" containerName="storage-initializer"
Sep 16 19:30:44.048262 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.048188    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df1e4bf-f13b-48ad-955c-c145483569a3" containerName="storage-initializer"
Sep 16 19:30:44.048262 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.048194    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df1e4bf-f13b-48ad-955c-c145483569a3" containerName="storage-initializer"
Sep 16 19:30:44.048262 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.048259    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df1e4bf-f13b-48ad-955c-c145483569a3" containerName="storage-initializer"
Sep 16 19:30:44.048262 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.048267    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df1e4bf-f13b-48ad-955c-c145483569a3" containerName="storage-initializer"
Sep 16 19:30:44.052933 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.052911    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.055052 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.055027    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 16 19:30:44.055215 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.055197    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:30:44.061842 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.061817    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"]
Sep 16 19:30:44.158519 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.158474    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/227e6de0-0678-4f5c-844a-66a98ea9249b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-fn2gk\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.158714 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.158537    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/227e6de0-0678-4f5c-844a-66a98ea9249b-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-fn2gk\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.158714 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.158596    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/227e6de0-0678-4f5c-844a-66a98ea9249b-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-fn2gk\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.158714 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.158657    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb74m\" (UniqueName: \"kubernetes.io/projected/227e6de0-0678-4f5c-844a-66a98ea9249b-kube-api-access-cb74m\") pod \"isvc-pmml-runtime-predictor-885fc8777-fn2gk\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.158714 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.158681    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/227e6de0-0678-4f5c-844a-66a98ea9249b-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-fn2gk\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.168031 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.167993    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 19:30:44.259118 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.259050    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/227e6de0-0678-4f5c-844a-66a98ea9249b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-fn2gk\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.259336 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.259140    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/227e6de0-0678-4f5c-844a-66a98ea9249b-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-fn2gk\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.259336 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.259165    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/227e6de0-0678-4f5c-844a-66a98ea9249b-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-fn2gk\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.259336 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.259208    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb74m\" (UniqueName: \"kubernetes.io/projected/227e6de0-0678-4f5c-844a-66a98ea9249b-kube-api-access-cb74m\") pod \"isvc-pmml-runtime-predictor-885fc8777-fn2gk\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.259336 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.259237    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/227e6de0-0678-4f5c-844a-66a98ea9249b-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-fn2gk\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.259662 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.259632    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/227e6de0-0678-4f5c-844a-66a98ea9249b-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-fn2gk\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.259925 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.259902    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/227e6de0-0678-4f5c-844a-66a98ea9249b-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-fn2gk\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.260074 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.260051    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/227e6de0-0678-4f5c-844a-66a98ea9249b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-fn2gk\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.261770 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.261752    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/227e6de0-0678-4f5c-844a-66a98ea9249b-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-fn2gk\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.268242 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.268217    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb74m\" (UniqueName: \"kubernetes.io/projected/227e6de0-0678-4f5c-844a-66a98ea9249b-kube-api-access-cb74m\") pod \"isvc-pmml-runtime-predictor-885fc8777-fn2gk\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.364177 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.364064    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:44.625663 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.625632    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df1e4bf-f13b-48ad-955c-c145483569a3" path="/var/lib/kubelet/pods/6df1e4bf-f13b-48ad-955c-c145483569a3/volumes"
Sep 16 19:30:44.645191 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:44.645154    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"]
Sep 16 19:30:44.647439 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:30:44.647384    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227e6de0_0678_4f5c_844a_66a98ea9249b.slice/crio-2b6f61bc517baa7665e0ed06175428ad4f425d32d3bf60867c228d04d00f8963 WatchSource:0}: Error finding container 2b6f61bc517baa7665e0ed06175428ad4f425d32d3bf60867c228d04d00f8963: Status 404 returned error can't find the container with id 2b6f61bc517baa7665e0ed06175428ad4f425d32d3bf60867c228d04d00f8963
Sep 16 19:30:45.252021 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:45.251986    2552 generic.go:356] "Generic (PLEG): container finished" podID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f" exitCode=1
Sep 16 19:30:45.252514 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:45.252061    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" event={"ID":"7297ae2a-6069-4a8a-aadb-9ebaf57ddc90","Type":"ContainerDied","Data":"341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"}
Sep 16 19:30:45.252514 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:45.252148    2552 scope.go:117] "RemoveContainer" containerID="fd9bfdabf8a8c78ec1a5baaddc8746663b8625936146ed22c2453023f08eb0f9"
Sep 16 19:30:45.252652 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:45.252531    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:30:45.252798 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:30:45.252768    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:30:45.254153 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:45.254130    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" event={"ID":"227e6de0-0678-4f5c-844a-66a98ea9249b","Type":"ContainerStarted","Data":"d89c2814608bbbe162018815b6185d53d8afaf970571b39c945059634ac45aea"}
Sep 16 19:30:45.254274 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:45.254181    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" event={"ID":"227e6de0-0678-4f5c-844a-66a98ea9249b","Type":"ContainerStarted","Data":"2b6f61bc517baa7665e0ed06175428ad4f425d32d3bf60867c228d04d00f8963"}
Sep 16 19:30:46.787975 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:46.787927    2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:30:46.788413 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:46.788330    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:30:46.788520 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:30:46.788499    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:30:49.276224 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:49.276187    2552 generic.go:356] "Generic (PLEG): container finished" podID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerID="d89c2814608bbbe162018815b6185d53d8afaf970571b39c945059634ac45aea" exitCode=0
Sep 16 19:30:49.276630 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:49.276259    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" event={"ID":"227e6de0-0678-4f5c-844a-66a98ea9249b","Type":"ContainerDied","Data":"d89c2814608bbbe162018815b6185d53d8afaf970571b39c945059634ac45aea"}
Sep 16 19:30:49.584122 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:49.583997    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq"
Sep 16 19:30:49.584431 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:49.584417    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:30:49.584617 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:30:49.584598    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:30:50.283097 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:50.283050    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" event={"ID":"227e6de0-0678-4f5c-844a-66a98ea9249b","Type":"ContainerStarted","Data":"fc417a5d1cd2467379f0b8e9ab4384f6d417c6d21d6445ab79a3692c3eaf5736"}
Sep 16 19:30:50.283507 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:50.283130    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" event={"ID":"227e6de0-0678-4f5c-844a-66a98ea9249b","Type":"ContainerStarted","Data":"1978e549cebdfa92bd2bebc943b3ddc6c164daced55d3f9dfcc9e7b3b9f1b5e1"}
Sep 16 19:30:50.283507 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:50.283328    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:50.303583 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:50.303532    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podStartSLOduration=2.349394906 podStartE2EDuration="6.303515611s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.954120705s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:30:50.301737245 +0000 UTC m=+3046.195924514" watchObservedRunningTime="2026-09-16 19:30:50.303515611 +0000 UTC m=+3046.197702880"
Sep 16 19:30:51.288352 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:51.288315    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:51.289807 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:51.289773    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 16 19:30:52.293175 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:52.293131    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 16 19:30:54.168078 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:54.168032    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 19:30:57.298266 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:57.298234    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:30:57.298911 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:30:57.298885    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 16 19:31:00.620498 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:00.620458    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:31:00.620992 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:31:00.620705    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:31:04.167921 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:04.167877    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 19:31:07.299585 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:07.299547    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 16 19:31:12.619783 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:12.619746    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:31:12.620261 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:31:12.619953    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:31:14.168752 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:14.168710    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 19:31:17.299211 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:17.299165    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 16 19:31:24.168727 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:24.168679    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 19:31:25.619722 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:25.619684    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:31:25.620146 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:31:25.619866    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:31:27.299194 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:27.299157    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 16 19:31:34.167997 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:34.167950    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 19:31:37.299592 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:37.299551    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 16 19:31:37.619965 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:37.619873    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:31:37.620163 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:31:37.620079    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:31:42.619923 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:42.619877    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 19:31:47.299382 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:47.299342    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 16 19:31:50.619825 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:50.619792    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:31:50.620235 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:31:50.619981    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:31:52.625713 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:52.625683    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:31:55.381803 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:55.381749    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"]
Sep 16 19:31:55.382411 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:55.382120    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kserve-container" containerID="cri-o://b4df2e0d77a4e554bf19f8a5b55dc36cbdf0173e6ad2d821027a01a720c4ec02" gracePeriod=30
Sep 16 19:31:55.382411 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:55.382132    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kube-rbac-proxy" containerID="cri-o://4bb450b8faaa6941a32ef065e0e9e180dab58e0c9669fbde6fb69949158d031f" gracePeriod=30
Sep 16 19:31:55.587427 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:55.587386    2552 generic.go:356] "Generic (PLEG): container finished" podID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerID="4bb450b8faaa6941a32ef065e0e9e180dab58e0c9669fbde6fb69949158d031f" exitCode=2
Sep 16 19:31:55.587611 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:55.587456    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" event={"ID":"64996fab-1c6f-4ae5-aa04-632ddd689846","Type":"ContainerDied","Data":"4bb450b8faaa6941a32ef065e0e9e180dab58e0c9669fbde6fb69949158d031f"}
Sep 16 19:31:57.299009 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:57.298964    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 16 19:31:59.161205 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.161164    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.64:8643/healthz\": dial tcp 10.132.0.64:8643: connect: connection refused"
Sep 16 19:31:59.329023 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.329000    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:31:59.394547 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.394474    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64996fab-1c6f-4ae5-aa04-632ddd689846-proxy-tls\") pod \"64996fab-1c6f-4ae5-aa04-632ddd689846\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") "
Sep 16 19:31:59.394681 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.394549    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64996fab-1c6f-4ae5-aa04-632ddd689846-kserve-provision-location\") pod \"64996fab-1c6f-4ae5-aa04-632ddd689846\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") "
Sep 16 19:31:59.394681 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.394595    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh8sz\" (UniqueName: \"kubernetes.io/projected/64996fab-1c6f-4ae5-aa04-632ddd689846-kube-api-access-nh8sz\") pod \"64996fab-1c6f-4ae5-aa04-632ddd689846\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") "
Sep 16 19:31:59.394681 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.394641    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64996fab-1c6f-4ae5-aa04-632ddd689846-kube-rbac-proxy-sar-config\") pod \"64996fab-1c6f-4ae5-aa04-632ddd689846\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") "
Sep 16 19:31:59.394681 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.394676    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64996fab-1c6f-4ae5-aa04-632ddd689846-cabundle-cert\") pod \"64996fab-1c6f-4ae5-aa04-632ddd689846\" (UID: \"64996fab-1c6f-4ae5-aa04-632ddd689846\") "
Sep 16 19:31:59.394937 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.394913    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64996fab-1c6f-4ae5-aa04-632ddd689846-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64996fab-1c6f-4ae5-aa04-632ddd689846" (UID: "64996fab-1c6f-4ae5-aa04-632ddd689846"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:59.395039 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.395010    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64996fab-1c6f-4ae5-aa04-632ddd689846-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "64996fab-1c6f-4ae5-aa04-632ddd689846" (UID: "64996fab-1c6f-4ae5-aa04-632ddd689846"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:59.395039 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.395032    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64996fab-1c6f-4ae5-aa04-632ddd689846-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64996fab-1c6f-4ae5-aa04-632ddd689846" (UID: "64996fab-1c6f-4ae5-aa04-632ddd689846"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:59.396510 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.396490    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64996fab-1c6f-4ae5-aa04-632ddd689846-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "64996fab-1c6f-4ae5-aa04-632ddd689846" (UID: "64996fab-1c6f-4ae5-aa04-632ddd689846"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:59.396635 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.396615    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64996fab-1c6f-4ae5-aa04-632ddd689846-kube-api-access-nh8sz" (OuterVolumeSpecName: "kube-api-access-nh8sz") pod "64996fab-1c6f-4ae5-aa04-632ddd689846" (UID: "64996fab-1c6f-4ae5-aa04-632ddd689846"). InnerVolumeSpecName "kube-api-access-nh8sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:59.496238 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.496199    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64996fab-1c6f-4ae5-aa04-632ddd689846-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:31:59.496238 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.496237    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh8sz\" (UniqueName: \"kubernetes.io/projected/64996fab-1c6f-4ae5-aa04-632ddd689846-kube-api-access-nh8sz\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:31:59.496410 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.496251    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64996fab-1c6f-4ae5-aa04-632ddd689846-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:31:59.496410 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.496266    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64996fab-1c6f-4ae5-aa04-632ddd689846-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:31:59.496410 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.496278    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64996fab-1c6f-4ae5-aa04-632ddd689846-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:31:59.607810 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.607778    2552 generic.go:356] "Generic (PLEG): container finished" podID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerID="b4df2e0d77a4e554bf19f8a5b55dc36cbdf0173e6ad2d821027a01a720c4ec02" exitCode=0
Sep 16 19:31:59.607919 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.607841    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"
Sep 16 19:31:59.607919 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.607861    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" event={"ID":"64996fab-1c6f-4ae5-aa04-632ddd689846","Type":"ContainerDied","Data":"b4df2e0d77a4e554bf19f8a5b55dc36cbdf0173e6ad2d821027a01a720c4ec02"}
Sep 16 19:31:59.607999 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.607918    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5" event={"ID":"64996fab-1c6f-4ae5-aa04-632ddd689846","Type":"ContainerDied","Data":"21342cf5716bf46364849d207fba9e5bfaf1d43b79d988c28ccddb7278b58285"}
Sep 16 19:31:59.607999 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.607947    2552 scope.go:117] "RemoveContainer" containerID="4bb450b8faaa6941a32ef065e0e9e180dab58e0c9669fbde6fb69949158d031f"
Sep 16 19:31:59.616576 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.616560    2552 scope.go:117] "RemoveContainer" containerID="b4df2e0d77a4e554bf19f8a5b55dc36cbdf0173e6ad2d821027a01a720c4ec02"
Sep 16 19:31:59.623758 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.623743    2552 scope.go:117] "RemoveContainer" containerID="9118325ce28d71d8c7130b617202c8e82093d9e361a0c5200f91607e7b3fa4d9"
Sep 16 19:31:59.631583 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.631563    2552 scope.go:117] "RemoveContainer" containerID="4bb450b8faaa6941a32ef065e0e9e180dab58e0c9669fbde6fb69949158d031f"
Sep 16 19:31:59.631873 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:31:59.631849    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb450b8faaa6941a32ef065e0e9e180dab58e0c9669fbde6fb69949158d031f\": container with ID starting with 4bb450b8faaa6941a32ef065e0e9e180dab58e0c9669fbde6fb69949158d031f not found: ID does not exist" containerID="4bb450b8faaa6941a32ef065e0e9e180dab58e0c9669fbde6fb69949158d031f"
Sep 16 19:31:59.631973 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.631883    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb450b8faaa6941a32ef065e0e9e180dab58e0c9669fbde6fb69949158d031f"} err="failed to get container status \"4bb450b8faaa6941a32ef065e0e9e180dab58e0c9669fbde6fb69949158d031f\": rpc error: code = NotFound desc = could not find container \"4bb450b8faaa6941a32ef065e0e9e180dab58e0c9669fbde6fb69949158d031f\": container with ID starting with 4bb450b8faaa6941a32ef065e0e9e180dab58e0c9669fbde6fb69949158d031f not found: ID does not exist"
Sep 16 19:31:59.631973 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.631906    2552 scope.go:117] "RemoveContainer" containerID="b4df2e0d77a4e554bf19f8a5b55dc36cbdf0173e6ad2d821027a01a720c4ec02"
Sep 16 19:31:59.632185 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:31:59.632170    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4df2e0d77a4e554bf19f8a5b55dc36cbdf0173e6ad2d821027a01a720c4ec02\": container with ID starting with b4df2e0d77a4e554bf19f8a5b55dc36cbdf0173e6ad2d821027a01a720c4ec02 not found: ID does not exist" containerID="b4df2e0d77a4e554bf19f8a5b55dc36cbdf0173e6ad2d821027a01a720c4ec02"
Sep 16 19:31:59.632266 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.632194    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4df2e0d77a4e554bf19f8a5b55dc36cbdf0173e6ad2d821027a01a720c4ec02"} err="failed to get container status \"b4df2e0d77a4e554bf19f8a5b55dc36cbdf0173e6ad2d821027a01a720c4ec02\": rpc error: code = NotFound desc = could not find container \"b4df2e0d77a4e554bf19f8a5b55dc36cbdf0173e6ad2d821027a01a720c4ec02\": container with ID starting with b4df2e0d77a4e554bf19f8a5b55dc36cbdf0173e6ad2d821027a01a720c4ec02 not found: ID does not exist"
Sep 16 19:31:59.632266 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.632214    2552 scope.go:117] "RemoveContainer" containerID="9118325ce28d71d8c7130b617202c8e82093d9e361a0c5200f91607e7b3fa4d9"
Sep 16 19:31:59.632500 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:31:59.632479    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9118325ce28d71d8c7130b617202c8e82093d9e361a0c5200f91607e7b3fa4d9\": container with ID starting with 9118325ce28d71d8c7130b617202c8e82093d9e361a0c5200f91607e7b3fa4d9 not found: ID does not exist" containerID="9118325ce28d71d8c7130b617202c8e82093d9e361a0c5200f91607e7b3fa4d9"
Sep 16 19:31:59.632578 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.632504    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9118325ce28d71d8c7130b617202c8e82093d9e361a0c5200f91607e7b3fa4d9"} err="failed to get container status \"9118325ce28d71d8c7130b617202c8e82093d9e361a0c5200f91607e7b3fa4d9\": rpc error: code = NotFound desc = could not find container \"9118325ce28d71d8c7130b617202c8e82093d9e361a0c5200f91607e7b3fa4d9\": container with ID starting with 9118325ce28d71d8c7130b617202c8e82093d9e361a0c5200f91607e7b3fa4d9 not found: ID does not exist"
Sep 16 19:31:59.632578 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.632545    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"]
Sep 16 19:31:59.640848 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:31:59.640827    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5"]
Sep 16 19:32:00.624666 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:00.624635    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" path="/var/lib/kubelet/pods/64996fab-1c6f-4ae5-aa04-632ddd689846/volumes"
Sep 16 19:32:04.623369 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:04.623317    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:32:04.623830 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:32:04.623519    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:32:07.299555 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:07.299514    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 16 19:32:08.620313 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:08.620265    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 16 19:32:16.619864 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:16.619821    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:32:16.620309 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:32:16.620070    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:32:18.624988 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:18.624952    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:32:25.160308 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:25.160267    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"]
Sep 16 19:32:25.160746 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:25.160571    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container" containerID="cri-o://1978e549cebdfa92bd2bebc943b3ddc6c164daced55d3f9dfcc9e7b3b9f1b5e1" gracePeriod=30
Sep 16 19:32:25.160746 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:25.160643    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kube-rbac-proxy" containerID="cri-o://fc417a5d1cd2467379f0b8e9ab4384f6d417c6d21d6445ab79a3692c3eaf5736" gracePeriod=30
Sep 16 19:32:25.725955 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:25.725922    2552 generic.go:356] "Generic (PLEG): container finished" podID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerID="fc417a5d1cd2467379f0b8e9ab4384f6d417c6d21d6445ab79a3692c3eaf5736" exitCode=2
Sep 16 19:32:25.726155 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:25.725969    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" event={"ID":"227e6de0-0678-4f5c-844a-66a98ea9249b","Type":"ContainerDied","Data":"fc417a5d1cd2467379f0b8e9ab4384f6d417c6d21d6445ab79a3692c3eaf5736"}
Sep 16 19:32:27.294016 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:27.293922    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.66:8643/healthz\": dial tcp 10.132.0.66:8643: connect: connection refused"
Sep 16 19:32:28.619706 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:28.619669    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:32:28.620124 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:32:28.619920    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:32:28.620323 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:28.620291    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 16 19:32:29.007468 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.007443    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:32:29.142429 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.142321    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/227e6de0-0678-4f5c-844a-66a98ea9249b-cabundle-cert\") pod \"227e6de0-0678-4f5c-844a-66a98ea9249b\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") "
Sep 16 19:32:29.142429 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.142378    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/227e6de0-0678-4f5c-844a-66a98ea9249b-kserve-provision-location\") pod \"227e6de0-0678-4f5c-844a-66a98ea9249b\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") "
Sep 16 19:32:29.142674 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.142451    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb74m\" (UniqueName: \"kubernetes.io/projected/227e6de0-0678-4f5c-844a-66a98ea9249b-kube-api-access-cb74m\") pod \"227e6de0-0678-4f5c-844a-66a98ea9249b\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") "
Sep 16 19:32:29.142674 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.142517    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/227e6de0-0678-4f5c-844a-66a98ea9249b-proxy-tls\") pod \"227e6de0-0678-4f5c-844a-66a98ea9249b\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") "
Sep 16 19:32:29.142674 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.142556    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/227e6de0-0678-4f5c-844a-66a98ea9249b-kube-rbac-proxy-sar-config\") pod \"227e6de0-0678-4f5c-844a-66a98ea9249b\" (UID: \"227e6de0-0678-4f5c-844a-66a98ea9249b\") "
Sep 16 19:32:29.142821 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.142766    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227e6de0-0678-4f5c-844a-66a98ea9249b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "227e6de0-0678-4f5c-844a-66a98ea9249b" (UID: "227e6de0-0678-4f5c-844a-66a98ea9249b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:29.142821 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.142791    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227e6de0-0678-4f5c-844a-66a98ea9249b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "227e6de0-0678-4f5c-844a-66a98ea9249b" (UID: "227e6de0-0678-4f5c-844a-66a98ea9249b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:29.142962 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.142937    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227e6de0-0678-4f5c-844a-66a98ea9249b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "227e6de0-0678-4f5c-844a-66a98ea9249b" (UID: "227e6de0-0678-4f5c-844a-66a98ea9249b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:29.144761 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.144725    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227e6de0-0678-4f5c-844a-66a98ea9249b-kube-api-access-cb74m" (OuterVolumeSpecName: "kube-api-access-cb74m") pod "227e6de0-0678-4f5c-844a-66a98ea9249b" (UID: "227e6de0-0678-4f5c-844a-66a98ea9249b"). InnerVolumeSpecName "kube-api-access-cb74m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:29.144761 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.144742    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227e6de0-0678-4f5c-844a-66a98ea9249b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "227e6de0-0678-4f5c-844a-66a98ea9249b" (UID: "227e6de0-0678-4f5c-844a-66a98ea9249b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:29.243992 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.243925    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/227e6de0-0678-4f5c-844a-66a98ea9249b-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:32:29.243992 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.243983    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/227e6de0-0678-4f5c-844a-66a98ea9249b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:32:29.243992 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.243997    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/227e6de0-0678-4f5c-844a-66a98ea9249b-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:32:29.243992 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.244010    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/227e6de0-0678-4f5c-844a-66a98ea9249b-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:32:29.244333 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.244024    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb74m\" (UniqueName: \"kubernetes.io/projected/227e6de0-0678-4f5c-844a-66a98ea9249b-kube-api-access-cb74m\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:32:29.746570 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.746535    2552 generic.go:356] "Generic (PLEG): container finished" podID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerID="1978e549cebdfa92bd2bebc943b3ddc6c164daced55d3f9dfcc9e7b3b9f1b5e1" exitCode=0
Sep 16 19:32:29.747067 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.746603    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" event={"ID":"227e6de0-0678-4f5c-844a-66a98ea9249b","Type":"ContainerDied","Data":"1978e549cebdfa92bd2bebc943b3ddc6c164daced55d3f9dfcc9e7b3b9f1b5e1"}
Sep 16 19:32:29.747067 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.746620    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"
Sep 16 19:32:29.747067 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.746669    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk" event={"ID":"227e6de0-0678-4f5c-844a-66a98ea9249b","Type":"ContainerDied","Data":"2b6f61bc517baa7665e0ed06175428ad4f425d32d3bf60867c228d04d00f8963"}
Sep 16 19:32:29.747067 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.746709    2552 scope.go:117] "RemoveContainer" containerID="fc417a5d1cd2467379f0b8e9ab4384f6d417c6d21d6445ab79a3692c3eaf5736"
Sep 16 19:32:29.755424 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.755407    2552 scope.go:117] "RemoveContainer" containerID="1978e549cebdfa92bd2bebc943b3ddc6c164daced55d3f9dfcc9e7b3b9f1b5e1"
Sep 16 19:32:29.762705 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.762687    2552 scope.go:117] "RemoveContainer" containerID="d89c2814608bbbe162018815b6185d53d8afaf970571b39c945059634ac45aea"
Sep 16 19:32:29.769937 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.769921    2552 scope.go:117] "RemoveContainer" containerID="fc417a5d1cd2467379f0b8e9ab4384f6d417c6d21d6445ab79a3692c3eaf5736"
Sep 16 19:32:29.770289 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:32:29.770271    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc417a5d1cd2467379f0b8e9ab4384f6d417c6d21d6445ab79a3692c3eaf5736\": container with ID starting with fc417a5d1cd2467379f0b8e9ab4384f6d417c6d21d6445ab79a3692c3eaf5736 not found: ID does not exist" containerID="fc417a5d1cd2467379f0b8e9ab4384f6d417c6d21d6445ab79a3692c3eaf5736"
Sep 16 19:32:29.770348 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.770295    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc417a5d1cd2467379f0b8e9ab4384f6d417c6d21d6445ab79a3692c3eaf5736"} err="failed to get container status \"fc417a5d1cd2467379f0b8e9ab4384f6d417c6d21d6445ab79a3692c3eaf5736\": rpc error: code = NotFound desc = could not find container \"fc417a5d1cd2467379f0b8e9ab4384f6d417c6d21d6445ab79a3692c3eaf5736\": container with ID starting with fc417a5d1cd2467379f0b8e9ab4384f6d417c6d21d6445ab79a3692c3eaf5736 not found: ID does not exist"
Sep 16 19:32:29.770348 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.770311    2552 scope.go:117] "RemoveContainer" containerID="1978e549cebdfa92bd2bebc943b3ddc6c164daced55d3f9dfcc9e7b3b9f1b5e1"
Sep 16 19:32:29.770606 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:32:29.770575    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1978e549cebdfa92bd2bebc943b3ddc6c164daced55d3f9dfcc9e7b3b9f1b5e1\": container with ID starting with 1978e549cebdfa92bd2bebc943b3ddc6c164daced55d3f9dfcc9e7b3b9f1b5e1 not found: ID does not exist" containerID="1978e549cebdfa92bd2bebc943b3ddc6c164daced55d3f9dfcc9e7b3b9f1b5e1"
Sep 16 19:32:29.770722 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.770613    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1978e549cebdfa92bd2bebc943b3ddc6c164daced55d3f9dfcc9e7b3b9f1b5e1"} err="failed to get container status \"1978e549cebdfa92bd2bebc943b3ddc6c164daced55d3f9dfcc9e7b3b9f1b5e1\": rpc error: code = NotFound desc = could not find container \"1978e549cebdfa92bd2bebc943b3ddc6c164daced55d3f9dfcc9e7b3b9f1b5e1\": container with ID starting with 1978e549cebdfa92bd2bebc943b3ddc6c164daced55d3f9dfcc9e7b3b9f1b5e1 not found: ID does not exist"
Sep 16 19:32:29.770722 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.770630    2552 scope.go:117] "RemoveContainer" containerID="d89c2814608bbbe162018815b6185d53d8afaf970571b39c945059634ac45aea"
Sep 16 19:32:29.770898 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:32:29.770869    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89c2814608bbbe162018815b6185d53d8afaf970571b39c945059634ac45aea\": container with ID starting with d89c2814608bbbe162018815b6185d53d8afaf970571b39c945059634ac45aea not found: ID does not exist" containerID="d89c2814608bbbe162018815b6185d53d8afaf970571b39c945059634ac45aea"
Sep 16 19:32:29.771008 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.770903    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89c2814608bbbe162018815b6185d53d8afaf970571b39c945059634ac45aea"} err="failed to get container status \"d89c2814608bbbe162018815b6185d53d8afaf970571b39c945059634ac45aea\": rpc error: code = NotFound desc = could not find container \"d89c2814608bbbe162018815b6185d53d8afaf970571b39c945059634ac45aea\": container with ID starting with d89c2814608bbbe162018815b6185d53d8afaf970571b39c945059634ac45aea not found: ID does not exist"
Sep 16 19:32:29.772928 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.772909    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"]
Sep 16 19:32:29.775764 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:29.775745    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk"]
Sep 16 19:32:30.418958 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.418926    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"]
Sep 16 19:32:30.419288 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419273    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kube-rbac-proxy"
Sep 16 19:32:30.419288 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419290    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kube-rbac-proxy"
Sep 16 19:32:30.419397 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419309    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kserve-container"
Sep 16 19:32:30.419397 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419315    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kserve-container"
Sep 16 19:32:30.419397 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419326    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="storage-initializer"
Sep 16 19:32:30.419397 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419333    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="storage-initializer"
Sep 16 19:32:30.419397 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419338    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container"
Sep 16 19:32:30.419397 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419343    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container"
Sep 16 19:32:30.419397 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419348    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kube-rbac-proxy"
Sep 16 19:32:30.419397 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419353    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kube-rbac-proxy"
Sep 16 19:32:30.419397 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419362    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="storage-initializer"
Sep 16 19:32:30.419397 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419367    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="storage-initializer"
Sep 16 19:32:30.419695 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419415    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kserve-container"
Sep 16 19:32:30.419695 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419423    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kube-rbac-proxy"
Sep 16 19:32:30.419695 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419429    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" containerName="kube-rbac-proxy"
Sep 16 19:32:30.419695 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.419435    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="64996fab-1c6f-4ae5-aa04-632ddd689846" containerName="kserve-container"
Sep 16 19:32:30.423965 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.423948    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.427512 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.427493    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:32:30.427624 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.427497    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:32:30.427624 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.427538    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:32:30.427624 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.427547    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:32:30.427846 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.427640    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:32:30.427846 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.427662    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 16 19:32:30.427846 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.427660    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 19:32:30.436954 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.436928    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"]
Sep 16 19:32:30.554419 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.554383    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af4f4c52-0aff-4ccd-9195-8792099e0568-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.554419 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.554422    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af4f4c52-0aff-4ccd-9195-8792099e0568-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.554630 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.554448    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z6zl\" (UniqueName: \"kubernetes.io/projected/af4f4c52-0aff-4ccd-9195-8792099e0568-kube-api-access-6z6zl\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.554630 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.554517    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af4f4c52-0aff-4ccd-9195-8792099e0568-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.554630 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.554542    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4f4c52-0aff-4ccd-9195-8792099e0568-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.626441 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.626400    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="227e6de0-0678-4f5c-844a-66a98ea9249b" path="/var/lib/kubelet/pods/227e6de0-0678-4f5c-844a-66a98ea9249b/volumes"
Sep 16 19:32:30.654998 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.654960    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4f4c52-0aff-4ccd-9195-8792099e0568-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.655162 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.655029    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af4f4c52-0aff-4ccd-9195-8792099e0568-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.655162 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.655053    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af4f4c52-0aff-4ccd-9195-8792099e0568-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.655162 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.655074    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z6zl\" (UniqueName: \"kubernetes.io/projected/af4f4c52-0aff-4ccd-9195-8792099e0568-kube-api-access-6z6zl\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.655334 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.655161    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af4f4c52-0aff-4ccd-9195-8792099e0568-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.655622 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.655596    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af4f4c52-0aff-4ccd-9195-8792099e0568-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.655838 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.655819    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af4f4c52-0aff-4ccd-9195-8792099e0568-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.655909 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.655848    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af4f4c52-0aff-4ccd-9195-8792099e0568-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.657519 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.657500    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4f4c52-0aff-4ccd-9195-8792099e0568-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.665449 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.665427    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z6zl\" (UniqueName: \"kubernetes.io/projected/af4f4c52-0aff-4ccd-9195-8792099e0568-kube-api-access-6z6zl\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:30.734153 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:30.734056    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:31.027817 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:31.027727    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"]
Sep 16 19:32:31.031963 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:32:31.031929    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4f4c52_0aff_4ccd_9195_8792099e0568.slice/crio-b6191d041e978a243cde7dfee69cceb200c0487cad8b7a8efa6c1ecb17cc3786 WatchSource:0}: Error finding container b6191d041e978a243cde7dfee69cceb200c0487cad8b7a8efa6c1ecb17cc3786: Status 404 returned error can't find the container with id b6191d041e978a243cde7dfee69cceb200c0487cad8b7a8efa6c1ecb17cc3786
Sep 16 19:32:31.759024 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:31.758986    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" event={"ID":"af4f4c52-0aff-4ccd-9195-8792099e0568","Type":"ContainerStarted","Data":"09aa2de82a930914d5786f7367ca8c0f148c6af369d656e2bbc7704e79ee6575"}
Sep 16 19:32:31.759211 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:31.759045    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" event={"ID":"af4f4c52-0aff-4ccd-9195-8792099e0568","Type":"ContainerStarted","Data":"b6191d041e978a243cde7dfee69cceb200c0487cad8b7a8efa6c1ecb17cc3786"}
Sep 16 19:32:35.779174 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:35.779137    2552 generic.go:356] "Generic (PLEG): container finished" podID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerID="09aa2de82a930914d5786f7367ca8c0f148c6af369d656e2bbc7704e79ee6575" exitCode=0
Sep 16 19:32:35.779644 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:35.779215    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" event={"ID":"af4f4c52-0aff-4ccd-9195-8792099e0568","Type":"ContainerDied","Data":"09aa2de82a930914d5786f7367ca8c0f148c6af369d656e2bbc7704e79ee6575"}
Sep 16 19:32:36.785675 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:36.785640    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" event={"ID":"af4f4c52-0aff-4ccd-9195-8792099e0568","Type":"ContainerStarted","Data":"5dca561cb3f9aee095a2e1563926d883356ee238f04bb0ec65a18d6f40951f1c"}
Sep 16 19:32:36.786127 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:36.785704    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" event={"ID":"af4f4c52-0aff-4ccd-9195-8792099e0568","Type":"ContainerStarted","Data":"8f667dfa89cb51606688f0a3cca12c5b1a28463ad6fa51e0a73c58245d71d284"}
Sep 16 19:32:36.786127 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:36.785972    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:36.786242 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:36.786126    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:36.787634 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:36.787609    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 16 19:32:37.789829 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:37.789794    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 16 19:32:41.620482 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:41.620447    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:32:41.620959 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:32:41.620638    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:32:42.794754 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:42.794723    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:32:42.795259 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:42.795231    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 16 19:32:42.820290 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:42.820223    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podStartSLOduration=8.744390052 podStartE2EDuration="12.820203606s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.075813554s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:36.805974067 +0000 UTC m=+3152.700161336" watchObservedRunningTime="2026-09-16 19:32:42.820203606 +0000 UTC m=+3158.714390878"
Sep 16 19:32:52.796072 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:52.796031    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 16 19:32:55.619819 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:32:55.619783    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:32:55.620260 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:32:55.619969    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:33:02.795831 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:33:02.795788    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 16 19:33:10.625153 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:33:10.625123    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:33:10.625538 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:33:10.625310    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:33:12.795297 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:33:12.795246    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 16 19:33:22.796072 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:33:22.796031    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 16 19:33:23.619967 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:33:23.619924    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:33:23.620241 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:33:23.620186    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:33:32.795555 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:33:32.795504    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 16 19:33:34.621535 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:33:34.621494    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:33:34.622018 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:33:34.621760    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:33:42.795289 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:33:42.795249    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 16 19:33:45.620239 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:33:45.620204    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:33:45.620733 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:33:45.620416    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:33:52.795543 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:33:52.795501    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 16 19:33:54.621915 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:33:54.621877    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 16 19:33:59.619905 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:33:59.619869    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:33:59.620325 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:33:59.620046    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:34:04.626149 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:04.626119    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:34:11.479229 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:11.479189    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"]
Sep 16 19:34:11.479729 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:11.479485    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container" containerID="cri-o://8f667dfa89cb51606688f0a3cca12c5b1a28463ad6fa51e0a73c58245d71d284" gracePeriod=30
Sep 16 19:34:11.479729 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:11.479568    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kube-rbac-proxy" containerID="cri-o://5dca561cb3f9aee095a2e1563926d883356ee238f04bb0ec65a18d6f40951f1c" gracePeriod=30
Sep 16 19:34:12.208911 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:12.208876    2552 generic.go:356] "Generic (PLEG): container finished" podID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerID="5dca561cb3f9aee095a2e1563926d883356ee238f04bb0ec65a18d6f40951f1c" exitCode=2
Sep 16 19:34:12.209105 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:12.208945    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" event={"ID":"af4f4c52-0aff-4ccd-9195-8792099e0568","Type":"ContainerDied","Data":"5dca561cb3f9aee095a2e1563926d883356ee238f04bb0ec65a18d6f40951f1c"}
Sep 16 19:34:12.790673 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:12.790629    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.67:8643/healthz\": dial tcp 10.132.0.67:8643: connect: connection refused"
Sep 16 19:34:13.620518 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:13.620479    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:34:13.620707 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:34:13.620684    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:34:14.621983 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:14.621937    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 16 19:34:15.419750 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.419720    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:34:15.517382 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.517292    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af4f4c52-0aff-4ccd-9195-8792099e0568-kube-rbac-proxy-sar-config\") pod \"af4f4c52-0aff-4ccd-9195-8792099e0568\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") "
Sep 16 19:34:15.517382 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.517333    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af4f4c52-0aff-4ccd-9195-8792099e0568-kserve-provision-location\") pod \"af4f4c52-0aff-4ccd-9195-8792099e0568\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") "
Sep 16 19:34:15.517382 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.517380    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af4f4c52-0aff-4ccd-9195-8792099e0568-cabundle-cert\") pod \"af4f4c52-0aff-4ccd-9195-8792099e0568\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") "
Sep 16 19:34:15.517672 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.517399    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z6zl\" (UniqueName: \"kubernetes.io/projected/af4f4c52-0aff-4ccd-9195-8792099e0568-kube-api-access-6z6zl\") pod \"af4f4c52-0aff-4ccd-9195-8792099e0568\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") "
Sep 16 19:34:15.517672 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.517425    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4f4c52-0aff-4ccd-9195-8792099e0568-proxy-tls\") pod \"af4f4c52-0aff-4ccd-9195-8792099e0568\" (UID: \"af4f4c52-0aff-4ccd-9195-8792099e0568\") "
Sep 16 19:34:15.517846 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.517704    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4f4c52-0aff-4ccd-9195-8792099e0568-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "af4f4c52-0aff-4ccd-9195-8792099e0568" (UID: "af4f4c52-0aff-4ccd-9195-8792099e0568"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:34:15.517846 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.517765    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4f4c52-0aff-4ccd-9195-8792099e0568-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "af4f4c52-0aff-4ccd-9195-8792099e0568" (UID: "af4f4c52-0aff-4ccd-9195-8792099e0568"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:15.517846 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.517778    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4f4c52-0aff-4ccd-9195-8792099e0568-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "af4f4c52-0aff-4ccd-9195-8792099e0568" (UID: "af4f4c52-0aff-4ccd-9195-8792099e0568"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:34:15.519529 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.519506    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4f4c52-0aff-4ccd-9195-8792099e0568-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af4f4c52-0aff-4ccd-9195-8792099e0568" (UID: "af4f4c52-0aff-4ccd-9195-8792099e0568"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:15.519736 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.519715    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4f4c52-0aff-4ccd-9195-8792099e0568-kube-api-access-6z6zl" (OuterVolumeSpecName: "kube-api-access-6z6zl") pod "af4f4c52-0aff-4ccd-9195-8792099e0568" (UID: "af4f4c52-0aff-4ccd-9195-8792099e0568"). InnerVolumeSpecName "kube-api-access-6z6zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:15.618126 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.618057    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af4f4c52-0aff-4ccd-9195-8792099e0568-cabundle-cert\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:34:15.618126 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.618124    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z6zl\" (UniqueName: \"kubernetes.io/projected/af4f4c52-0aff-4ccd-9195-8792099e0568-kube-api-access-6z6zl\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:34:15.618345 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.618135    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4f4c52-0aff-4ccd-9195-8792099e0568-proxy-tls\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:34:15.618345 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.618148    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af4f4c52-0aff-4ccd-9195-8792099e0568-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:34:15.618345 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:15.618157    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af4f4c52-0aff-4ccd-9195-8792099e0568-kserve-provision-location\") on node \"ip-10-0-139-72.ec2.internal\" DevicePath \"\""
Sep 16 19:34:16.229114 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.229058    2552 generic.go:356] "Generic (PLEG): container finished" podID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerID="8f667dfa89cb51606688f0a3cca12c5b1a28463ad6fa51e0a73c58245d71d284" exitCode=0
Sep 16 19:34:16.229569 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.229145    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"
Sep 16 19:34:16.229569 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.229143    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" event={"ID":"af4f4c52-0aff-4ccd-9195-8792099e0568","Type":"ContainerDied","Data":"8f667dfa89cb51606688f0a3cca12c5b1a28463ad6fa51e0a73c58245d71d284"}
Sep 16 19:34:16.229569 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.229297    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8" event={"ID":"af4f4c52-0aff-4ccd-9195-8792099e0568","Type":"ContainerDied","Data":"b6191d041e978a243cde7dfee69cceb200c0487cad8b7a8efa6c1ecb17cc3786"}
Sep 16 19:34:16.229569 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.229340    2552 scope.go:117] "RemoveContainer" containerID="5dca561cb3f9aee095a2e1563926d883356ee238f04bb0ec65a18d6f40951f1c"
Sep 16 19:34:16.238819 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.238793    2552 scope.go:117] "RemoveContainer" containerID="8f667dfa89cb51606688f0a3cca12c5b1a28463ad6fa51e0a73c58245d71d284"
Sep 16 19:34:16.248056 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.248033    2552 scope.go:117] "RemoveContainer" containerID="09aa2de82a930914d5786f7367ca8c0f148c6af369d656e2bbc7704e79ee6575"
Sep 16 19:34:16.253532 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.253490    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"]
Sep 16 19:34:16.256567 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.256536    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8"]
Sep 16 19:34:16.257727 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.257709    2552 scope.go:117] "RemoveContainer" containerID="5dca561cb3f9aee095a2e1563926d883356ee238f04bb0ec65a18d6f40951f1c"
Sep 16 19:34:16.258097 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:34:16.258067    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dca561cb3f9aee095a2e1563926d883356ee238f04bb0ec65a18d6f40951f1c\": container with ID starting with 5dca561cb3f9aee095a2e1563926d883356ee238f04bb0ec65a18d6f40951f1c not found: ID does not exist" containerID="5dca561cb3f9aee095a2e1563926d883356ee238f04bb0ec65a18d6f40951f1c"
Sep 16 19:34:16.258190 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.258113    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dca561cb3f9aee095a2e1563926d883356ee238f04bb0ec65a18d6f40951f1c"} err="failed to get container status \"5dca561cb3f9aee095a2e1563926d883356ee238f04bb0ec65a18d6f40951f1c\": rpc error: code = NotFound desc = could not find container \"5dca561cb3f9aee095a2e1563926d883356ee238f04bb0ec65a18d6f40951f1c\": container with ID starting with 5dca561cb3f9aee095a2e1563926d883356ee238f04bb0ec65a18d6f40951f1c not found: ID does not exist"
Sep 16 19:34:16.258190 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.258134    2552 scope.go:117] "RemoveContainer" containerID="8f667dfa89cb51606688f0a3cca12c5b1a28463ad6fa51e0a73c58245d71d284"
Sep 16 19:34:16.258441 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:34:16.258421    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f667dfa89cb51606688f0a3cca12c5b1a28463ad6fa51e0a73c58245d71d284\": container with ID starting with 8f667dfa89cb51606688f0a3cca12c5b1a28463ad6fa51e0a73c58245d71d284 not found: ID does not exist" containerID="8f667dfa89cb51606688f0a3cca12c5b1a28463ad6fa51e0a73c58245d71d284"
Sep 16 19:34:16.258487 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.258448    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f667dfa89cb51606688f0a3cca12c5b1a28463ad6fa51e0a73c58245d71d284"} err="failed to get container status \"8f667dfa89cb51606688f0a3cca12c5b1a28463ad6fa51e0a73c58245d71d284\": rpc error: code = NotFound desc = could not find container \"8f667dfa89cb51606688f0a3cca12c5b1a28463ad6fa51e0a73c58245d71d284\": container with ID starting with 8f667dfa89cb51606688f0a3cca12c5b1a28463ad6fa51e0a73c58245d71d284 not found: ID does not exist"
Sep 16 19:34:16.258487 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.258465    2552 scope.go:117] "RemoveContainer" containerID="09aa2de82a930914d5786f7367ca8c0f148c6af369d656e2bbc7704e79ee6575"
Sep 16 19:34:16.258727 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:34:16.258710    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09aa2de82a930914d5786f7367ca8c0f148c6af369d656e2bbc7704e79ee6575\": container with ID starting with 09aa2de82a930914d5786f7367ca8c0f148c6af369d656e2bbc7704e79ee6575 not found: ID does not exist" containerID="09aa2de82a930914d5786f7367ca8c0f148c6af369d656e2bbc7704e79ee6575"
Sep 16 19:34:16.258784 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.258730    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09aa2de82a930914d5786f7367ca8c0f148c6af369d656e2bbc7704e79ee6575"} err="failed to get container status \"09aa2de82a930914d5786f7367ca8c0f148c6af369d656e2bbc7704e79ee6575\": rpc error: code = NotFound desc = could not find container \"09aa2de82a930914d5786f7367ca8c0f148c6af369d656e2bbc7704e79ee6575\": container with ID starting with 09aa2de82a930914d5786f7367ca8c0f148c6af369d656e2bbc7704e79ee6575 not found: ID does not exist"
Sep 16 19:34:16.625430 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:16.625397    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" path="/var/lib/kubelet/pods/af4f4c52-0aff-4ccd-9195-8792099e0568/volumes"
Sep 16 19:34:28.620254 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:28.620216    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:34:28.620691 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:34:28.620394    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:34:41.620512 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:41.620472    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:34:41.620888 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:34:41.620628    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:34:53.620574 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:53.620539    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:34:53.621044 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:34:53.620718    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:34:56.056975 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:56.056948    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g5lz9_8058be60-2358-4acd-bc5b-6382f4fa2e3e/global-pull-secret-syncer/0.log"
Sep 16 19:34:56.249907 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:56.249875    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9phx7_cddd1ee3-5425-435b-9607-8d9f2ea8d836/konnectivity-agent/0.log"
Sep 16 19:34:56.367990 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:34:56.367908    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-72.ec2.internal_8b97e76fa4a597a2b9f96eb244b741d8/haproxy/0.log"
Sep 16 19:35:01.089011 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:01.088979    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-54b8f9ff88-lkgg8_bbe0642e-3461-40d3-bee5-6be6545fc3d1/metrics-server/0.log"
Sep 16 19:35:01.120018 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:01.119988    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-kbwnp_d67daef5-60ea-49a2-8a86-a6bf94dcebbc/monitoring-plugin/0.log"
Sep 16 19:35:01.326607 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:01.326574    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qj79k_032307cc-4c2d-4ff7-9be6-4b9baadd54fb/node-exporter/0.log"
Sep 16 19:35:01.354382 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:01.354321    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qj79k_032307cc-4c2d-4ff7-9be6-4b9baadd54fb/kube-rbac-proxy/0.log"
Sep 16 19:35:01.380776 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:01.380752    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qj79k_032307cc-4c2d-4ff7-9be6-4b9baadd54fb/init-textfile/0.log"
Sep 16 19:35:01.415826 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:01.415798    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-r5c45_e47c5d54-4b22-4722-bb8c-b1ef622ac42f/kube-rbac-proxy-main/0.log"
Sep 16 19:35:01.442050 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:01.442024    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-r5c45_e47c5d54-4b22-4722-bb8c-b1ef622ac42f/kube-rbac-proxy-self/0.log"
Sep 16 19:35:01.473046 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:01.473015    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-r5c45_e47c5d54-4b22-4722-bb8c-b1ef622ac42f/openshift-state-metrics/0.log"
Sep 16 19:35:01.733422 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:01.733397    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-rxzfb_842b872c-6600-4b43-9ca0-8ae13cd550d8/prometheus-operator/0.log"
Sep 16 19:35:01.761072 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:01.761045    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-rxzfb_842b872c-6600-4b43-9ca0-8ae13cd550d8/kube-rbac-proxy/0.log"
Sep 16 19:35:01.791563 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:01.791541    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-pmwbl_745a1716-da85-4e62-a4d4-bb680e50cc86/prometheus-operator-admission-webhook/0.log"
Sep 16 19:35:01.837986 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:01.837942    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75cc585756-x9qwr_36f7055b-0255-4a0f-a1f6-971cae2af884/telemeter-client/0.log"
Sep 16 19:35:01.872958 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:01.872932    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75cc585756-x9qwr_36f7055b-0255-4a0f-a1f6-971cae2af884/reload/0.log"
Sep 16 19:35:01.901306 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:01.901285    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75cc585756-x9qwr_36f7055b-0255-4a0f-a1f6-971cae2af884/kube-rbac-proxy/0.log"
Sep 16 19:35:03.828191 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.828152    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"]
Sep 16 19:35:03.828595 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.828507    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container"
Sep 16 19:35:03.828595 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.828519    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container"
Sep 16 19:35:03.828595 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.828534    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kube-rbac-proxy"
Sep 16 19:35:03.828595 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.828540    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kube-rbac-proxy"
Sep 16 19:35:03.828595 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.828559    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="storage-initializer"
Sep 16 19:35:03.828595 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.828568    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="storage-initializer"
Sep 16 19:35:03.828802 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.828626    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kube-rbac-proxy"
Sep 16 19:35:03.828802 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.828637    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="af4f4c52-0aff-4ccd-9195-8792099e0568" containerName="kserve-container"
Sep 16 19:35:03.831619 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.831597    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:03.834162 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.834130    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vps5\"/\"openshift-service-ca.crt\""
Sep 16 19:35:03.834961 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.834936    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vps5\"/\"kube-root-ca.crt\""
Sep 16 19:35:03.834961 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.834947    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2vps5\"/\"default-dockercfg-j62vv\""
Sep 16 19:35:03.842417 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.842386    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"]
Sep 16 19:35:03.936005 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.935969    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a48ee9b-a924-4508-9413-0f346e599953-lib-modules\") pod \"perf-node-gather-daemonset-gkhs7\" (UID: \"3a48ee9b-a924-4508-9413-0f346e599953\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:03.936005 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.936008    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a48ee9b-a924-4508-9413-0f346e599953-podres\") pod \"perf-node-gather-daemonset-gkhs7\" (UID: \"3a48ee9b-a924-4508-9413-0f346e599953\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:03.936304 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.936033    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95gkm\" (UniqueName: \"kubernetes.io/projected/3a48ee9b-a924-4508-9413-0f346e599953-kube-api-access-95gkm\") pod \"perf-node-gather-daemonset-gkhs7\" (UID: \"3a48ee9b-a924-4508-9413-0f346e599953\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:03.936304 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.936138    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a48ee9b-a924-4508-9413-0f346e599953-sys\") pod \"perf-node-gather-daemonset-gkhs7\" (UID: \"3a48ee9b-a924-4508-9413-0f346e599953\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:03.936304 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.936198    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a48ee9b-a924-4508-9413-0f346e599953-proc\") pod \"perf-node-gather-daemonset-gkhs7\" (UID: \"3a48ee9b-a924-4508-9413-0f346e599953\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:03.995128 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:03.995067    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-648d6c6bd5-tgvzq_2eff5a46-e5d8-4e51-956d-09843653654e/console/0.log"
Sep 16 19:35:04.037543 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:04.037501    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95gkm\" (UniqueName: \"kubernetes.io/projected/3a48ee9b-a924-4508-9413-0f346e599953-kube-api-access-95gkm\") pod \"perf-node-gather-daemonset-gkhs7\" (UID: \"3a48ee9b-a924-4508-9413-0f346e599953\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:04.037760 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:04.037556    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a48ee9b-a924-4508-9413-0f346e599953-sys\") pod \"perf-node-gather-daemonset-gkhs7\" (UID: \"3a48ee9b-a924-4508-9413-0f346e599953\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:04.037760 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:04.037596    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a48ee9b-a924-4508-9413-0f346e599953-proc\") pod \"perf-node-gather-daemonset-gkhs7\" (UID: \"3a48ee9b-a924-4508-9413-0f346e599953\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:04.037760 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:04.037671    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a48ee9b-a924-4508-9413-0f346e599953-lib-modules\") pod \"perf-node-gather-daemonset-gkhs7\" (UID: \"3a48ee9b-a924-4508-9413-0f346e599953\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:04.037760 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:04.037694    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a48ee9b-a924-4508-9413-0f346e599953-sys\") pod \"perf-node-gather-daemonset-gkhs7\" (UID: \"3a48ee9b-a924-4508-9413-0f346e599953\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:04.037760 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:04.037699    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a48ee9b-a924-4508-9413-0f346e599953-podres\") pod \"perf-node-gather-daemonset-gkhs7\" (UID: \"3a48ee9b-a924-4508-9413-0f346e599953\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:04.037970 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:04.037778    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a48ee9b-a924-4508-9413-0f346e599953-proc\") pod \"perf-node-gather-daemonset-gkhs7\" (UID: \"3a48ee9b-a924-4508-9413-0f346e599953\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:04.037970 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:04.037831    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a48ee9b-a924-4508-9413-0f346e599953-lib-modules\") pod \"perf-node-gather-daemonset-gkhs7\" (UID: \"3a48ee9b-a924-4508-9413-0f346e599953\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:04.037970 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:04.037813    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a48ee9b-a924-4508-9413-0f346e599953-podres\") pod \"perf-node-gather-daemonset-gkhs7\" (UID: \"3a48ee9b-a924-4508-9413-0f346e599953\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:04.048732 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:04.048702    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95gkm\" (UniqueName: \"kubernetes.io/projected/3a48ee9b-a924-4508-9413-0f346e599953-kube-api-access-95gkm\") pod \"perf-node-gather-daemonset-gkhs7\" (UID: \"3a48ee9b-a924-4508-9413-0f346e599953\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:04.142940 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:04.142908    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:04.413773 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:04.413666    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"]
Sep 16 19:35:04.417199 ip-10-0-139-72 kubenswrapper[2552]: W0916 19:35:04.417167    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3a48ee9b_a924_4508_9413_0f346e599953.slice/crio-9d8295ec190210275283a9ab279bf0d261a470f49f11f67574130e332b1399aa WatchSource:0}: Error finding container 9d8295ec190210275283a9ab279bf0d261a470f49f11f67574130e332b1399aa: Status 404 returned error can't find the container with id 9d8295ec190210275283a9ab279bf0d261a470f49f11f67574130e332b1399aa
Sep 16 19:35:04.448829 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:04.448800    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7" event={"ID":"3a48ee9b-a924-4508-9413-0f346e599953","Type":"ContainerStarted","Data":"9d8295ec190210275283a9ab279bf0d261a470f49f11f67574130e332b1399aa"}
Sep 16 19:35:05.454225 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:05.454184    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7" event={"ID":"3a48ee9b-a924-4508-9413-0f346e599953","Type":"ContainerStarted","Data":"146fe6e5b0fdcf53453408204c9e0d4df4c74c12aeb83ab4571acd6eb0b623f9"}
Sep 16 19:35:05.454657 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:05.454281    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:05.472577 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:05.472515    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7" podStartSLOduration=2.47249799 podStartE2EDuration="2.47249799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:05.471636178 +0000 UTC m=+3301.365823447" watchObservedRunningTime="2026-09-16 19:35:05.47249799 +0000 UTC m=+3301.366685258"
Sep 16 19:35:05.650317 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:05.650287    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cs4tf_083ee546-65d4-43b1-9cb4-383989984db0/dns/0.log"
Sep 16 19:35:05.677741 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:05.677719    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cs4tf_083ee546-65d4-43b1-9cb4-383989984db0/kube-rbac-proxy/0.log"
Sep 16 19:35:05.832674 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:05.832588    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tw7bx_be0cf19b-10fb-41c7-be6b-efb9c89b566e/dns-node-resolver/0.log"
Sep 16 19:35:06.423188 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:06.423156    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z4qmj_aa40654a-058d-4aab-8eeb-b34d81ac854b/node-ca/0.log"
Sep 16 19:35:06.620581 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:06.620546    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:35:06.620962 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:35:06.620758    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:35:07.141001 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:07.140969    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bd67b9f94-9bddk_d9a7b520-0ba2-428b-9927-d671e633215a/router/0.log"
Sep 16 19:35:07.513039 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:07.512955    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bhkzx_0f31a781-91cf-40f7-94e7-6560c4d94cfa/serve-healthcheck-canary/0.log"
Sep 16 19:35:07.952190 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:07.952160    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8z6x_cc51025e-e317-4422-8196-00e87137b6f1/kube-rbac-proxy/0.log"
Sep 16 19:35:07.976580 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:07.976557    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8z6x_cc51025e-e317-4422-8196-00e87137b6f1/exporter/0.log"
Sep 16 19:35:08.001233 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:08.001209    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8z6x_cc51025e-e317-4422-8196-00e87137b6f1/extractor/0.log"
Sep 16 19:35:10.634876 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:10.634841    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-mvmcq_7297ae2a-6069-4a8a-aadb-9ebaf57ddc90/manager/9.log"
Sep 16 19:35:10.634876 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:10.634863    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-mvmcq_7297ae2a-6069-4a8a-aadb-9ebaf57ddc90/manager/9.log"
Sep 16 19:35:10.933958 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:10.933874    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-f9dg5_64085754-3c26-43b2-b8cf-60ac4140c852/manager/0.log"
Sep 16 19:35:11.126251 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:11.126221    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-d8qkl_dd615c57-c80e-4893-906d-355e4aa634cf/seaweedfs/0.log"
Sep 16 19:35:11.183174 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:11.183149    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-vcrv9_1a810dd1-218b-4452-bf08-0ddb51fca795/seaweedfs-tls-serving/0.log"
Sep 16 19:35:11.468967 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:11.468939    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-gkhs7"
Sep 16 19:35:17.340038 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:17.340008    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rnccg_346b4c37-63f2-42b6-989f-14bcff205a35/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:35:17.379616 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:17.379577    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rnccg_346b4c37-63f2-42b6-989f-14bcff205a35/egress-router-binary-copy/0.log"
Sep 16 19:35:17.421389 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:17.421368    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rnccg_346b4c37-63f2-42b6-989f-14bcff205a35/cni-plugins/0.log"
Sep 16 19:35:17.452067 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:17.452038    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rnccg_346b4c37-63f2-42b6-989f-14bcff205a35/bond-cni-plugin/0.log"
Sep 16 19:35:17.489522 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:17.489500    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rnccg_346b4c37-63f2-42b6-989f-14bcff205a35/routeoverride-cni/0.log"
Sep 16 19:35:17.526522 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:17.526497    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rnccg_346b4c37-63f2-42b6-989f-14bcff205a35/whereabouts-cni-bincopy/0.log"
Sep 16 19:35:17.570580 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:17.570551    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rnccg_346b4c37-63f2-42b6-989f-14bcff205a35/whereabouts-cni/0.log"
Sep 16 19:35:17.906030 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:17.906002    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zvqnt_3b830482-d9ca-4032-a057-2f87226eed51/kube-multus/0.log"
Sep 16 19:35:18.030178 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:18.030152    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zkgpv_0f5f1c97-1813-4d89-8af7-ce7d0eebc10c/network-metrics-daemon/0.log"
Sep 16 19:35:18.054400 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:18.054372    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zkgpv_0f5f1c97-1813-4d89-8af7-ce7d0eebc10c/kube-rbac-proxy/0.log"
Sep 16 19:35:19.077720 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:19.077684    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cs8zv_db04d8f0-efe5-4d33-871b-8a2421f26e17/ovn-controller/0.log"
Sep 16 19:35:19.097903 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:19.097871    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cs8zv_db04d8f0-efe5-4d33-871b-8a2421f26e17/ovn-acl-logging/0.log"
Sep 16 19:35:19.129227 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:19.129198    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cs8zv_db04d8f0-efe5-4d33-871b-8a2421f26e17/kube-rbac-proxy-node/0.log"
Sep 16 19:35:19.164949 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:19.164919    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cs8zv_db04d8f0-efe5-4d33-871b-8a2421f26e17/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:35:19.193661 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:19.193632    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cs8zv_db04d8f0-efe5-4d33-871b-8a2421f26e17/northd/0.log"
Sep 16 19:35:19.221103 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:19.221065    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cs8zv_db04d8f0-efe5-4d33-871b-8a2421f26e17/nbdb/0.log"
Sep 16 19:35:19.249745 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:19.249710    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cs8zv_db04d8f0-efe5-4d33-871b-8a2421f26e17/sbdb/0.log"
Sep 16 19:35:19.313104 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:19.313061    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cs8zv_db04d8f0-efe5-4d33-871b-8a2421f26e17/ovnkube-controller/0.log"
Sep 16 19:35:20.589719 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:20.589693    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qbtmz_18bae23f-10bf-41df-a14b-ba453b1ab8f1/network-check-target-container/0.log"
Sep 16 19:35:20.620139 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:20.620107    2552 scope.go:117] "RemoveContainer" containerID="341080fc6f0308590408360e7420d7143ae9cf3cfbeaa7058a6c48d383fe4c7f"
Sep 16 19:35:20.620339 ip-10-0-139-72 kubenswrapper[2552]: E0916 19:35:20.620311    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=llmisvc-controller-manager-645f9c8fd6-mvmcq_kserve(7297ae2a-6069-4a8a-aadb-9ebaf57ddc90)\"" pod="kserve/llmisvc-controller-manager-645f9c8fd6-mvmcq" podUID="7297ae2a-6069-4a8a-aadb-9ebaf57ddc90"
Sep 16 19:35:21.571210 ip-10-0-139-72 kubenswrapper[2552]: I0916 19:35:21.571178    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-jzjwf_55e5f454-d5af-40ab-a53f-b0e12313348b/iptables-alerter/0.log"