Sep 16 17:24:31.112186 ip-10-0-128-31 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 17:24:31.538628 ip-10-0-128-31 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 17:24:31.538628 ip-10-0-128-31 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 17:24:31.538628 ip-10-0-128-31 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 17:24:31.538628 ip-10-0-128-31 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 17:24:31.538628 ip-10-0-128-31 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 17:24:31.539316 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.538658    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 17:24:31.544239 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544223    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:31.544239 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544237    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:31.544239 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544241    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544244    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544247    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544250    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544254    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544257    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544259    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544262    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544266    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544268    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544271    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544273    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544280    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544282    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544285    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544288    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544290    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544293    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544295    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544297    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:31.544344 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544300    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544303    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544306    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544308    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544311    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544313    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544316    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544318    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544321    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544324    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544327    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544329    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544332    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544335    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544337    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544340    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544343    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544346    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544349    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544351    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:31.544807 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544354    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544357    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544360    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544362    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544365    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544367    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544371    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544373    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544376    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544378    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544381    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544383    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544386    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544389    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544391    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544394    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544397    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544400    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544404    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:31.545302 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544408    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544412    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544416    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544419    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544422    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544424    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544427    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544429    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544432    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544434    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544438    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544443    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544447    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544449    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544452    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544455    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544457    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544460    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544462    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:31.545742 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544465    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544467    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544469    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544472    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544475    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544477    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544480    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544482    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544485    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544487    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544490    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544492    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544495    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544498    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544500    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544503    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544506    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544509    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.544511    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545232    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:31.546201 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545239    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545242    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545245    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545247    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545256    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545258    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545261    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545264    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545267    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545269    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545272    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545275    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545277    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545280    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545282    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545285    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545287    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545290    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545292    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545295    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:31.546661 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545297    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545300    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545304    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545307    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545309    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545312    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545315    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545318    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545321    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545324    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545326    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545329    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545333    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545337    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545340    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545343    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545346    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545354    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545356    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545359    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:31.547157 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545361    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545364    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545367    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545369    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545372    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545374    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545377    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545380    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545383    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545386    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545389    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545392    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545395    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545397    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545400    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545403    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545406    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545409    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545411    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:31.547641 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545414    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545417    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545420    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545422    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545426    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545428    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545431    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545433    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545436    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545438    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545441    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545445    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545448    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545451    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545454    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545456    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545459    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545462    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545465    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:31.548121 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545468    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545470    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545473    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545476    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545479    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545482    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545485    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545488    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545490    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545493    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545496    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545498    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545501    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545503    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545506    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545508    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545511    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545513    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545516    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545518    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:31.548558 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545918    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545925    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545928    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545931    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545934    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545937    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545940    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545943    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545947    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545950    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545953    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545957    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545960    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545963    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545966    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545968    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545971    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545974    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545977    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:31.549065 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545980    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545983    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545986    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545988    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545991    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545994    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.545997    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546000    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546003    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546005    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546008    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546010    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546013    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546015    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546018    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546020    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546022    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546025    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546029    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546033    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:31.549518 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546035    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546038    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546040    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546043    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546046    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546048    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546051    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546053    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546055    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546058    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546060    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546063    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546066    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546068    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546070    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546073    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546076    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546079    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546081    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:31.549993 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546083    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546086    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546088    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546091    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546093    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546096    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546098    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546101    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546104    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546107    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546109    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546111    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546114    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546117    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546119    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546121    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546124    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546126    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546129    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546131    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:31.550463 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546134    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546136    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546138    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546141    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546143    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546146    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546148    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546151    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546154    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546157    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546159    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546163    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546167    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546170    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546172    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546175    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546178    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546181    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546183    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546185    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:31.550942 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546188    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546257    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546287    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546293    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546297    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546302    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546305    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546312    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546316    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546320    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546322    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546326    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546329    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546332    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546335    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546338    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546341    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546344    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546347    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546351    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546354    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546357    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546360    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546363    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 17:24:31.551476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546368    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546371    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546374    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546377    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546380    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546383    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546386    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546389    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546392    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546395    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546398    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546401    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546404    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546407    2581 flags.go:64] FLAG: --enable-server="true"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546411    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546415    2581 flags.go:64] FLAG: --event-burst="100"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546418    2581 flags.go:64] FLAG: --event-qps="50"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546421    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546424    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546427    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546430    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546433    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546436    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546439    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546442    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 17:24:31.552042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546445    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546448    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546451    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546454    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546456    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546459    2581 flags.go:64] FLAG: --feature-gates=""
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546465    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546468    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546471    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546475    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546478    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546480    2581 flags.go:64] FLAG: --help="false"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546483    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-128-31.ec2.internal"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546486    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546489    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546492    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546495    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546498    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546501    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546503    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546506    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546509    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546512    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546515    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 17:24:31.552641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546518    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546520    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546525    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546528    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546531    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546534    2581 flags.go:64] FLAG: --lock-file=""
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546536    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546539    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546542    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546548    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546550    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546553    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546556    2581 flags.go:64] FLAG: --logging-format="text"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546559    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546563    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546566    2581 flags.go:64] FLAG: --manifest-url=""
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546569    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546574    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546577    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546581    2581 flags.go:64] FLAG: --max-pods="110"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546583    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546587    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546590    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546593    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546596    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 17:24:31.553254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546598    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546601    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546609    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546612    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546614    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546617    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546621    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546626    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546629    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546632    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546636    2581 flags.go:64] FLAG: --port="10250"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546639    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546642    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-001b0c2a8f43b8a7e"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546645    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546648    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546651    2581 flags.go:64] FLAG: --register-node="true"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546654    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546657    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546660    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546663    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546665    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546670    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546673    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546676    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546679    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 17:24:31.554370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546682    2581 flags.go:64] FLAG: --runonce="false"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546685    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546688    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546691    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546694    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546696    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546699    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546702    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546705    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546708    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546710    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546713    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546716    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546719    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546722    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546725    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546730    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546734    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546737    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546741    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546744    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546747    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546749    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546752    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546755    2581 flags.go:64] FLAG: --v="2"
Sep 16 17:24:31.555435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546759    2581 flags.go:64] FLAG: --version="false"
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546763    2581 flags.go:64] FLAG: --vmodule=""
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546766    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.546771    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546901    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546905    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546908    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546911    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546914    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546917    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546920    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546923    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546925    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546929    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546931    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546934    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546937    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546939    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546941    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546944    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:31.556508 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546947    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546949    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546952    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546955    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546959    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546961    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546964    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546967    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546969    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546972    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546975    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546978    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546980    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546983    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546985    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546989    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546992    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546995    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.546997    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547000    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:31.557477 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547002    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547004    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547007    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547009    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547012    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547016    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547019    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547023    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547028    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547031    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547039    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547043    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547046    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547048    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547051    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547054    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547058    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547061    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547063    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:31.558214 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547066    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547068    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547071    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547074    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547076    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547079    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547081    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547084    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547088    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547091    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547093    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547096    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547099    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547102    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547104    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547107    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547110    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547113    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547115    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:31.558703 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547118    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547120    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547123    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547125    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547128    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547130    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547133    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547135    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547139    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547142    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547147    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547150    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547153    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547156    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547159    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547162    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547165    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547168    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547170    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:31.559485 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547173    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:31.560274 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547176    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:31.560274 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547180    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:31.560274 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547183    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:31.560274 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547186    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:31.560274 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.547188    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:31.560274 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.547193    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 17:24:31.560274 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.556237    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 17:24:31.560274 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.556418    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 17:24:31.560274 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.556447    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 17:24:31.560274 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.556456    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 17:24:31.560274 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556549    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:31.560274 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556557    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:31.560274 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556562    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:31.560274 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556566    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:31.560274 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556571    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556575    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556579    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556583    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556587    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556592    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556597    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556601    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556606    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556609    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556614    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556618    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556622    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556627    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556631    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556635    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556640    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556644    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556648    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556652    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:31.560639 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556656    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556660    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556664    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556668    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556673    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556685    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556690    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556695    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556699    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556703    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556707    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556711    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556715    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556719    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556723    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556727    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556732    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556738    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556744    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556748    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:31.561301 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556753    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556757    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556761    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556765    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556769    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556773    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556777    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556781    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556785    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556789    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556793    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556797    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556801    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556805    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556809    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556813    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556818    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556822    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556834    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556838    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:31.561783 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556842    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556849    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556855    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556859    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556863    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556868    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556872    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556875    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556879    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556883    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556910    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556915    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556919    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556922    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556926    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556930    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556936    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556942    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556946    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:31.562418 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556950    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:31.562870 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556954    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:31.562870 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556957    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:31.562870 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556960    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:31.562870 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556964    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:31.562870 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556967    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:31.562870 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556971    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:31.562870 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556974    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:31.562870 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556979    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:31.562870 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556983    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:31.562870 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556987    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:31.562870 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.556991    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:31.562870 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557002    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:31.562870 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557007    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:31.562870 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557011    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:31.562870 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557015    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:31.563338 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.557023    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 17:24:31.563338 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557255    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:31.563338 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557264    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:31.563338 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557269    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:31.563338 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557274    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:31.563338 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557278    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:31.563338 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557283    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:31.563338 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557287    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:31.563338 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557292    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:31.563338 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557296    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:31.563338 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557300    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:31.563338 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557304    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:31.563338 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557308    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:31.563338 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557312    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:31.563338 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557316    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557321    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557326    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557330    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557334    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557339    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557343    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557347    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557351    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557357    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557363    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557368    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557373    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557378    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557381    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557403    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557408    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557412    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557416    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557420    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:31.563705 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557424    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557428    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557432    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557436    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557440    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557444    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557448    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557453    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557456    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557460    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557464    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557468    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557472    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557475    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557479    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557483    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557487    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557491    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557495    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:31.564222 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557501    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557506    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557511    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557516    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557520    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557525    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557529    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557533    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557538    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557549    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557554    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557558    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557562    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557566    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557570    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557574    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557578    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557582    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557588    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:31.564663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557593    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557598    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557602    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557606    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557610    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557614    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557618    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557622    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557627    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557631    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557635    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557639    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557643    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557647    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557651    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557655    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557659    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557663    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557667    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557671    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:31.565420 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557675    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:31.566232 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557679    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:31.566232 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557683    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:31.566232 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557695    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:31.566232 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557699    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:31.566232 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557703    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:31.566232 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557707    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:31.566232 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:31.557711    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:31.566232 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.557718    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 17:24:31.566232 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.557920    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 17:24:31.566232 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.560962    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 17:24:31.566232 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.562115    2581 server.go:1019] "Starting client certificate rotation"
Sep 16 17:24:31.566232 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.562217    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 17:24:31.566232 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.562253    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 17:24:31.566809 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.565488    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 17:24:31.573075 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.573050    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 17:24:31.586271 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.586251    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 16 17:24:31.589714 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.589699    2581 log.go:25] "Validated CRI v1 image API"
Sep 16 17:24:31.590564 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.590551    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 17:24:31.594786 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.594762    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b82286b1-173b-45c7-a914-86de09393479:/dev/nvme0n1p4 d4569c75-ddce-482d-a338-1322b9d88e99:/dev/nvme0n1p3]
Sep 16 17:24:31.594858 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.594786    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 17:24:31.600354 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.600230    2581 manager.go:216] Machine: {Timestamp:2026-09-16 17:24:31.598839824 +0000 UTC m=+0.387462352 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3272217 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2428295964fce316daf589461fba61 SystemUUID:ec242829-5964-fce3-16da-f589461fba61 BootID:e181a73e-c152-4919-8403-a0cb5994dc1a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:be:55:07 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:be:55:07 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:92:2f:bf:5a:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 17:24:31.600354 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.600348    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 17:24:31.600474 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.600462    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 17:24:31.603943 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.603912    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 17:24:31.604123 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.603948    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-31.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 17:24:31.604927 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.604915    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 17:24:31.604987 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.604934    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 17:24:31.604987 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.604966    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 17:24:31.604987 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.604972    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 17:24:31.605903 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.605876    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 17:24:31.607115 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.607103    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 17:24:31.612599 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.612584    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 17:24:31.612925 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.612913    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 17:24:31.612987 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.612956    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 17:24:31.612987 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.612972    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 17:24:31.613080 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.612992    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 17:24:31.613080 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.613007    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 16 17:24:31.613080 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.613020    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 17:24:31.613439 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.613424    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 17:24:31.613559 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.613452    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 17:24:31.617602 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.617578    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 17:24:31.620524 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.620509    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 17:24:31.620583 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.620540    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 17:24:31.622766 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.622753    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 17:24:31.622828 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.622769    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 17:24:31.622828 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.622776    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 17:24:31.622828 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.622781    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 17:24:31.622828 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.622790    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 17:24:31.622828 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.622795    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 17:24:31.622828 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.622808    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 17:24:31.622828 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.622814    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 17:24:31.622828 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.622821    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 17:24:31.622828 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.622826    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 17:24:31.622828 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.622835    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 17:24:31.623216 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.622840    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 17:24:31.623779 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.623769    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 17:24:31.623779 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.623778    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 17:24:31.626916 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.626881    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 17:24:31.627001 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.626985    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-31.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 17:24:31.627137 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.627114    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 17:24:31.627753 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.627740    2581 server.go:1268] "Started kubelet"
Sep 16 17:24:31.627796 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.627772    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 17:24:31.627929 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.627864    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 17:24:31.627993 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.627956    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 17:24:31.628564 ip-10-0-128-31 systemd[1]: Started Kubernetes Kubelet.
Sep 16 17:24:31.632034 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.629488    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 17:24:31.632958 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.632942    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 17:24:31.633384 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.633364    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 17:24:31.638443 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.638426    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 17:24:31.638546 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.638439    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 17:24:31.638598 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.638579    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Sep 16 17:24:31.639346 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.639329    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 17:24:31.639346 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.639339    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 17:24:31.639483 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.639365    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 17:24:31.639483 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.639469    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 17:24:31.639483 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.639479    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 17:24:31.639739 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.639725    2581 factory.go:153] Registering CRI-O factory
Sep 16 17:24:31.639789 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.639743    2581 factory.go:223] Registration of the crio container factory successfully
Sep 16 17:24:31.639789 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.639783    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 17:24:31.639844 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.639790    2581 factory.go:55] Registering systemd factory
Sep 16 17:24:31.639844 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.639795    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 16 17:24:31.639844 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.639808    2581 factory.go:103] Registering Raw factory
Sep 16 17:24:31.639844 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.639816    2581 manager.go:1196] Started watching for new ooms in manager
Sep 16 17:24:31.640136 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.640125    2581 manager.go:349] Starting recovery of all containers
Sep 16 17:24:31.649748 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.649729    2581 manager.go:354] Recovery completed
Sep 16 17:24:31.651586 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.651559    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 17:24:31.652670 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.652597    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 17:24:31.652961 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.651462    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-31.ec2.internal.18d5dd0c3fb2fa4f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-31.ec2.internal,UID:ip-10-0-128-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-31.ec2.internal,},FirstTimestamp:2026-09-16 17:24:31.627680335 +0000 UTC m=+0.416302859,LastTimestamp:2026-09-16 17:24:31.627680335 +0000 UTC m=+0.416302859,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-31.ec2.internal,}"
Sep 16 17:24:31.654078 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.654067    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:31.656058 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.656035    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:31.656122 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.656072    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:31.656122 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.656082    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:31.656984 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.656967    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 17:24:31.656984 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.656981    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 17:24:31.657103 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.657002    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 17:24:31.657103 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.657005    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:31.657776 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.657762    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:31.657821 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.657789    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:31.657821 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.657803    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:31.659153 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.659140    2581 policy_none.go:49] "None policy: Start"
Sep 16 17:24:31.659210 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.659159    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 17:24:31.659210 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.659172    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 17:24:31.661172 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.661158    2581 policy_none.go:47] "Start"
Sep 16 17:24:31.662273 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.662210    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-31.ec2.internal.18d5dd0c4163ff19  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-31.ec2.internal,UID:ip-10-0-128-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-31.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-31.ec2.internal,},FirstTimestamp:2026-09-16 17:24:31.656058649 +0000 UTC m=+0.444681173,LastTimestamp:2026-09-16 17:24:31.656058649 +0000 UTC m=+0.444681173,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-31.ec2.internal,}"
Sep 16 17:24:31.670416 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.670355    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-31.ec2.internal.18d5dd0c4164474d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-31.ec2.internal,UID:ip-10-0-128-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-31.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-31.ec2.internal,},FirstTimestamp:2026-09-16 17:24:31.656077133 +0000 UTC m=+0.444699657,LastTimestamp:2026-09-16 17:24:31.656077133 +0000 UTC m=+0.444699657,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-31.ec2.internal,}"
Sep 16 17:24:31.687453 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.687383    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-31.ec2.internal.18d5dd0c41646b54  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-31.ec2.internal,UID:ip-10-0-128-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-31.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-31.ec2.internal,},FirstTimestamp:2026-09-16 17:24:31.656086356 +0000 UTC m=+0.444708881,LastTimestamp:2026-09-16 17:24:31.656086356 +0000 UTC m=+0.444708881,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-31.ec2.internal,}"
Sep 16 17:24:31.695294 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.695234    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-31.ec2.internal.18d5dd0c4163ff19\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-31.ec2.internal.18d5dd0c4163ff19  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-31.ec2.internal,UID:ip-10-0-128-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-31.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-31.ec2.internal,},FirstTimestamp:2026-09-16 17:24:31.656058649 +0000 UTC m=+0.444681173,LastTimestamp:2026-09-16 17:24:31.657775595 +0000 UTC m=+0.446398118,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-31.ec2.internal,}"
Sep 16 17:24:31.719718 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.699811    2581 manager.go:337] "Starting Device Plugin manager"
Sep 16 17:24:31.719718 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.699858    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 17:24:31.719718 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.699871    2581 server.go:87] "Starting device plugin registration server"
Sep 16 17:24:31.719718 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.700135    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 17:24:31.719718 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.700149    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 17:24:31.719718 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.700252    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 17:24:31.719718 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.700347    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 17:24:31.719718 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.700361    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 17:24:31.719718 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.701018    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 17:24:31.719718 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.701066    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-31.ec2.internal\" not found"
Sep 16 17:24:31.719718 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.715197    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-31.ec2.internal.18d5dd0c4164474d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-31.ec2.internal.18d5dd0c4164474d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-31.ec2.internal,UID:ip-10-0-128-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-31.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-31.ec2.internal,},FirstTimestamp:2026-09-16 17:24:31.656077133 +0000 UTC m=+0.444699657,LastTimestamp:2026-09-16 17:24:31.657795732 +0000 UTC m=+0.446418261,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-31.ec2.internal,}"
Sep 16 17:24:31.726276 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.726209    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-31.ec2.internal.18d5dd0c41646b54\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-31.ec2.internal.18d5dd0c41646b54  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-31.ec2.internal,UID:ip-10-0-128-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-31.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-31.ec2.internal,},FirstTimestamp:2026-09-16 17:24:31.656086356 +0000 UTC m=+0.444708881,LastTimestamp:2026-09-16 17:24:31.65780666 +0000 UTC m=+0.446429184,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-31.ec2.internal,}"
Sep 16 17:24:31.747305 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.747232    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-31.ec2.internal.18d5dd0c450abba0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-31.ec2.internal,UID:ip-10-0-128-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-128-31.ec2.internal,},FirstTimestamp:2026-09-16 17:24:31.717317536 +0000 UTC m=+0.505940047,LastTimestamp:2026-09-16 17:24:31.717317536 +0000 UTC m=+0.505940047,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-31.ec2.internal,}"
Sep 16 17:24:31.759129 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.759090    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 17:24:31.760436 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.760417    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 17:24:31.760542 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.760462    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 17:24:31.760542 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.760485    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 17:24:31.760542 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.760520    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 17:24:31.775163 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.775136    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 16 17:24:31.779599 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.779583    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pn255"
Sep 16 17:24:31.790501 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.790434    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pn255"
Sep 16 17:24:31.800764 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.800747    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:31.801700 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.801682    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:31.801789 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.801713    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:31.801789 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.801722    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:31.801789 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.801743    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-31.ec2.internal"
Sep 16 17:24:31.821403 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.821383    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-31.ec2.internal"
Sep 16 17:24:31.821476 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.821406    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-31.ec2.internal\": node \"ip-10-0-128-31.ec2.internal\" not found"
Sep 16 17:24:31.843793 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.843766    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Sep 16 17:24:31.861061 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.861025    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"]
Sep 16 17:24:31.861128 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.861108    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:31.863244 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.863225    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:31.863346 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.863261    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:31.863346 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.863293    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:31.865506 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.865494    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:31.865643 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.865627    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Sep 16 17:24:31.865745 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.865658    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:31.866413 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.866399    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:31.866483 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.866430    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:31.866483 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.866441    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:31.866483 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.866400    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:31.866577 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.866500    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:31.866577 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.866509    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:31.868644 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.868625    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Sep 16 17:24:31.868699 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.868653    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:31.869507 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.869491    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:31.869577 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.869521    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:31.869577 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.869536    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:31.892733 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.892705    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-31.ec2.internal\" not found" node="ip-10-0-128-31.ec2.internal"
Sep 16 17:24:31.896989 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.896971    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-31.ec2.internal\" not found" node="ip-10-0-128-31.ec2.internal"
Sep 16 17:24:31.940678 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.940651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a009e1b07a5529d711c19429f741484-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"4a009e1b07a5529d711c19429f741484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Sep 16 17:24:31.940801 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.940680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a009e1b07a5529d711c19429f741484-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"4a009e1b07a5529d711c19429f741484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Sep 16 17:24:31.940801 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:31.940697    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c40b89e27f34846eb7c4becaeceb3ee5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" (UID: \"c40b89e27f34846eb7c4becaeceb3ee5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Sep 16 17:24:31.944771 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:31.944752    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Sep 16 17:24:32.041343 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.041259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a009e1b07a5529d711c19429f741484-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"4a009e1b07a5529d711c19429f741484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Sep 16 17:24:32.041343 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.041285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a009e1b07a5529d711c19429f741484-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"4a009e1b07a5529d711c19429f741484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Sep 16 17:24:32.041343 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.041302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c40b89e27f34846eb7c4becaeceb3ee5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" (UID: \"c40b89e27f34846eb7c4becaeceb3ee5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Sep 16 17:24:32.041343 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.041343    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c40b89e27f34846eb7c4becaeceb3ee5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" (UID: \"c40b89e27f34846eb7c4becaeceb3ee5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Sep 16 17:24:32.041590 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.041372    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a009e1b07a5529d711c19429f741484-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"4a009e1b07a5529d711c19429f741484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Sep 16 17:24:32.041590 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.041375    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a009e1b07a5529d711c19429f741484-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"4a009e1b07a5529d711c19429f741484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Sep 16 17:24:32.045366 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:32.045350    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Sep 16 17:24:32.146149 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:32.146099    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Sep 16 17:24:32.196294 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.196259    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Sep 16 17:24:32.199608 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.199587    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Sep 16 17:24:32.246775 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:32.246738    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Sep 16 17:24:32.347358 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:32.347286    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Sep 16 17:24:32.447803 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:32.447771    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Sep 16 17:24:32.548383 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:32.548354    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Sep 16 17:24:32.561527 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.561505    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 17:24:32.638009 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.637937    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:32.648757 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:32.648731    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Sep 16 17:24:32.710188 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:32.709974    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a009e1b07a5529d711c19429f741484.slice/crio-c1c0585333b6c120bf0d84f0fce35c01afff5aa300e2be7b84f7277cdbb955c8 WatchSource:0}: Error finding container c1c0585333b6c120bf0d84f0fce35c01afff5aa300e2be7b84f7277cdbb955c8: Status 404 returned error can't find the container with id c1c0585333b6c120bf0d84f0fce35c01afff5aa300e2be7b84f7277cdbb955c8
Sep 16 17:24:32.710722 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:32.710701    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40b89e27f34846eb7c4becaeceb3ee5.slice/crio-8aaf3368bed0278f57a047493b3e6bb548824254bd262813683e13a13fcbbbf5 WatchSource:0}: Error finding container 8aaf3368bed0278f57a047493b3e6bb548824254bd262813683e13a13fcbbbf5: Status 404 returned error can't find the container with id 8aaf3368bed0278f57a047493b3e6bb548824254bd262813683e13a13fcbbbf5
Sep 16 17:24:32.714022 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.714008    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:24:32.749208 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:32.749186    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Sep 16 17:24:32.764355 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.764309    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" event={"ID":"4a009e1b07a5529d711c19429f741484","Type":"ContainerStarted","Data":"c1c0585333b6c120bf0d84f0fce35c01afff5aa300e2be7b84f7277cdbb955c8"}
Sep 16 17:24:32.765414 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.765393    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" event={"ID":"c40b89e27f34846eb7c4becaeceb3ee5","Type":"ContainerStarted","Data":"8aaf3368bed0278f57a047493b3e6bb548824254bd262813683e13a13fcbbbf5"}
Sep 16 17:24:32.794577 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.794540    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 17:19:32 +0000 UTC" deadline="2028-06-21 13:49:20.290869538 +0000 UTC"
Sep 16 17:24:32.794577 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.794574    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15452h24m47.496298397s"
Sep 16 17:24:32.827756 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.827719    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:32.828855 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.828832    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:32.828961 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.828867    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:32.828961 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.828877    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:32.849411 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:32.849392    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Sep 16 17:24:32.908365 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.908297    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:32.939326 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.939294    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Sep 16 17:24:32.942078 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.942061    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:32.958129 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.956390    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 17:24:32.959200 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.959183    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Sep 16 17:24:32.970439 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:32.970420    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 17:24:33.336359 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.336270    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:33.614493 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.614407    2581 apiserver.go:52] "Watching apiserver"
Sep 16 17:24:33.623615 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.623581    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 17:24:33.624012 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.623980    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-bvnzw","openshift-dns/node-resolver-7rg26","openshift-multus/multus-additional-cni-plugins-nn9tr","openshift-network-diagnostics/network-check-target-ppgtg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4","openshift-image-registry/node-ca-qzm9m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal","openshift-multus/multus-ncvfq","openshift-multus/network-metrics-daemon-jhf5t","openshift-network-operator/iptables-alerter-749nw","openshift-ovn-kubernetes/ovnkube-node-5kpv5","kube-system/global-pull-secret-syncer-jxft6","kube-system/konnectivity-agent-dlwht","kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"]
Sep 16 17:24:33.626863 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.626834    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-749nw"
Sep 16 17:24:33.629175 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.629137    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dlwht"
Sep 16 17:24:33.632039 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.632017    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 17:24:33.632039 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.632036    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 17:24:33.632256 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.632044    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ljxbp\""
Sep 16 17:24:33.632256 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.632093    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:24:33.632256 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.632160    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 17:24:33.632256 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.632221    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.633886 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.633751    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 17:24:33.633886 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.633852    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-85bfj\""
Sep 16 17:24:33.634616 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.634596    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:33.634704 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:33.634673    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:24:33.636716 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.636697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.638990 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.638963    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 17:24:33.639119 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.639101    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qzm9m"
Sep 16 17:24:33.640920 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.640883    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 17:24:33.640920 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.640883    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 17:24:33.641077 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.640887    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 17:24:33.641137 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.641078    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 17:24:33.641214 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.641199    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 17:24:33.641263 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.641237    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x8cfd\""
Sep 16 17:24:33.641529 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.641510    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:33.641603 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:33.641579    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:24:33.643621 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.643597    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 17:24:33.643999 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.643980    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 17:24:33.644293 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.644272    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 17:24:33.644496 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.644474    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k5jh9\""
Sep 16 17:24:33.644707 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.644690    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 17:24:33.644898 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.644863    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 17:24:33.645224 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.645207    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 17:24:33.645381 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.645368    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-npd6k\""
Sep 16 17:24:33.645529 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.645515    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.647919 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.647855    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7rg26"
Sep 16 17:24:33.650542 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.650523    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.651448 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.651547 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.651547 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgk85\" (UniqueName: \"kubernetes.io/projected/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-kube-api-access-pgk85\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.651547 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651535    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5638697b-24e8-4230-857d-7add1666f79a-host\") pod \"node-ca-qzm9m\" (UID: \"5638697b-24e8-4230-857d-7add1666f79a\") " pod="openshift-image-registry/node-ca-qzm9m"
Sep 16 17:24:33.651696 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651560    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pvkj\" (UniqueName: \"kubernetes.io/projected/5638697b-24e8-4230-857d-7add1666f79a-kube-api-access-5pvkj\") pod \"node-ca-qzm9m\" (UID: \"5638697b-24e8-4230-857d-7add1666f79a\") " pod="openshift-image-registry/node-ca-qzm9m"
Sep 16 17:24:33.651696 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651619    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-cnibin\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.651696 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs\") pod \"network-metrics-daemon-jhf5t\" (UID: \"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8\") " pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:33.651844 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgkw9\" (UniqueName: \"kubernetes.io/projected/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-kube-api-access-kgkw9\") pod \"network-metrics-daemon-jhf5t\" (UID: \"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8\") " pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:33.651844 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gkr9\" (UniqueName: \"kubernetes.io/projected/fe144550-4e3b-4c9a-8068-4dd0d03bd209-kube-api-access-9gkr9\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.651844 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4a459e7e-1c73-4170-b753-22882c81fa6d-konnectivity-ca\") pod \"konnectivity-agent-dlwht\" (UID: \"4a459e7e-1c73-4170-b753-22882c81fa6d\") " pod="kube-system/konnectivity-agent-dlwht"
Sep 16 17:24:33.651844 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hkhb\" (UniqueName: \"kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb\") pod \"network-check-target-ppgtg\" (UID: \"bde6ac75-b65f-4477-829e-4e37b5c55b19\") " pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:33.651844 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651826    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-socket-dir\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.652115 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-registration-dir\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.652115 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-sys-fs\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.652115 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5638697b-24e8-4230-857d-7add1666f79a-serviceca\") pod \"node-ca-qzm9m\" (UID: \"5638697b-24e8-4230-857d-7add1666f79a\") " pod="openshift-image-registry/node-ca-qzm9m"
Sep 16 17:24:33.652115 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.651980    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-system-cni-dir\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.652115 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.652024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4a459e7e-1c73-4170-b753-22882c81fa6d-agent-certs\") pod \"konnectivity-agent-dlwht\" (UID: \"4a459e7e-1c73-4170-b753-22882c81fa6d\") " pod="kube-system/konnectivity-agent-dlwht"
Sep 16 17:24:33.652115 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.652056    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.652115 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.652103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3cd98fb6-5344-470a-a91b-c491ba24c89a-iptables-alerter-script\") pod \"iptables-alerter-749nw\" (UID: \"3cd98fb6-5344-470a-a91b-c491ba24c89a\") " pod="openshift-network-operator/iptables-alerter-749nw"
Sep 16 17:24:33.652456 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.652132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3cd98fb6-5344-470a-a91b-c491ba24c89a-host-slash\") pod \"iptables-alerter-749nw\" (UID: \"3cd98fb6-5344-470a-a91b-c491ba24c89a\") " pod="openshift-network-operator/iptables-alerter-749nw"
Sep 16 17:24:33.652456 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.652153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zzbq\" (UniqueName: \"kubernetes.io/projected/3cd98fb6-5344-470a-a91b-c491ba24c89a-kube-api-access-7zzbq\") pod \"iptables-alerter-749nw\" (UID: \"3cd98fb6-5344-470a-a91b-c491ba24c89a\") " pod="openshift-network-operator/iptables-alerter-749nw"
Sep 16 17:24:33.652456 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.652169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-cni-binary-copy\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.652456 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.652191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.652456 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.652217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-device-dir\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.652456 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.652239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-etc-selinux\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.653047 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.653026    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 17:24:33.653135 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.653119    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mqrlw\""
Sep 16 17:24:33.653552 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.653493    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:24:33.653552 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.653498    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 17:24:33.653908 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.653798    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-98scm\""
Sep 16 17:24:33.653908 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.653816    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rw48l\""
Sep 16 17:24:33.653908 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.653849    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 17:24:33.655528 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.655503    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.658766 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.657525    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:33.658766 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:33.657601    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:24:33.660749 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.660621    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 17:24:33.660749 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.660650    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 17:24:33.661412 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.661394    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 17:24:33.661412 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.661405    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 17:24:33.661568 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.661448    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 17:24:33.661619 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.661581    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 17:24:33.661833 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.661811    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 17:24:33.661833 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.661829    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f5ts9\""
Sep 16 17:24:33.661955 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.661849    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 17:24:33.724793 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.724759    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2fdlb"
Sep 16 17:24:33.738311 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.738278    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2fdlb"
Sep 16 17:24:33.741111 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.741084    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 17:24:33.752610 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.752575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-var-lib-kubelet\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.752747 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.752627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgkw9\" (UniqueName: \"kubernetes.io/projected/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-kube-api-access-kgkw9\") pod \"network-metrics-daemon-jhf5t\" (UID: \"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8\") " pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:33.752819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.752774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-log-socket\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.752819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.752803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/50a056f9-6c58-4c10-91e6-4c8202e3cc20-env-overrides\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.752915 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.752831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80100b19-7ac7-4382-af81-3f0a97bd7072-dbus\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:33.752915 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.752861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gkr9\" (UniqueName: \"kubernetes.io/projected/fe144550-4e3b-4c9a-8068-4dd0d03bd209-kube-api-access-9gkr9\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.752915 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.752880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hkhb\" (UniqueName: \"kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb\") pod \"network-check-target-ppgtg\" (UID: \"bde6ac75-b65f-4477-829e-4e37b5c55b19\") " pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:33.752915 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.752895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5638697b-24e8-4230-857d-7add1666f79a-serviceca\") pod \"node-ca-qzm9m\" (UID: \"5638697b-24e8-4230-857d-7add1666f79a\") " pod="openshift-image-registry/node-ca-qzm9m"
Sep 16 17:24:33.753122 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.752938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-system-cni-dir\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.753122 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.752996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-system-cni-dir\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.753122 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-multus-socket-dir-parent\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.753122 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjpjm\" (UniqueName: \"kubernetes.io/projected/22254a42-55b7-4718-abf3-475178a65fc9-kube-api-access-vjpjm\") pod \"node-resolver-7rg26\" (UID: \"22254a42-55b7-4718-abf3-475178a65fc9\") " pod="openshift-dns/node-resolver-7rg26"
Sep 16 17:24:33.753293 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-run-systemd\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.753293 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3cd98fb6-5344-470a-a91b-c491ba24c89a-iptables-alerter-script\") pod \"iptables-alerter-749nw\" (UID: \"3cd98fb6-5344-470a-a91b-c491ba24c89a\") " pod="openshift-network-operator/iptables-alerter-749nw"
Sep 16 17:24:33.753293 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3cd98fb6-5344-470a-a91b-c491ba24c89a-host-slash\") pod \"iptables-alerter-749nw\" (UID: \"3cd98fb6-5344-470a-a91b-c491ba24c89a\") " pod="openshift-network-operator/iptables-alerter-749nw"
Sep 16 17:24:33.753293 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-var-lib-cni-bin\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.753293 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-etc-kubernetes\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.753293 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753261    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-lib-modules\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.753591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-slash\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.753591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753295    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3cd98fb6-5344-470a-a91b-c491ba24c89a-host-slash\") pod \"iptables-alerter-749nw\" (UID: \"3cd98fb6-5344-470a-a91b-c491ba24c89a\") " pod="openshift-network-operator/iptables-alerter-749nw"
Sep 16 17:24:33.753591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-cni-bin\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.753591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.753591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.753591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753445    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgk85\" (UniqueName: \"kubernetes.io/projected/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-kube-api-access-pgk85\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.753591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753455    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5638697b-24e8-4230-857d-7add1666f79a-serviceca\") pod \"node-ca-qzm9m\" (UID: \"5638697b-24e8-4230-857d-7add1666f79a\") " pod="openshift-image-registry/node-ca-qzm9m"
Sep 16 17:24:33.753591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-cni-binary-copy\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.753591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-var-lib-cni-multus\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.753591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-tuned\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.753591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7c29\" (UniqueName: \"kubernetes.io/projected/50a056f9-6c58-4c10-91e6-4c8202e3cc20-kube-api-access-c7c29\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.753591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5638697b-24e8-4230-857d-7add1666f79a-host\") pod \"node-ca-qzm9m\" (UID: \"5638697b-24e8-4230-857d-7add1666f79a\") " pod="openshift-image-registry/node-ca-qzm9m"
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-cnibin\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs\") pod \"network-metrics-daemon-jhf5t\" (UID: \"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8\") " pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-cnibin\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753703    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-kubernetes\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-run-openvswitch\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3cd98fb6-5344-470a-a91b-c491ba24c89a-iptables-alerter-script\") pod \"iptables-alerter-749nw\" (UID: \"3cd98fb6-5344-470a-a91b-c491ba24c89a\") " pod="openshift-network-operator/iptables-alerter-749nw"
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/50a056f9-6c58-4c10-91e6-4c8202e3cc20-ovnkube-script-lib\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-socket-dir\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:33.753821    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-sys-fs\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:33.753891    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs podName:fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:34.253870811 +0000 UTC m=+3.042493346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs") pod "network-metrics-daemon-jhf5t" (UID: "fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-kubelet\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.753957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5638697b-24e8-4230-857d-7add1666f79a-host\") pod \"node-ca-qzm9m\" (UID: \"5638697b-24e8-4230-857d-7add1666f79a\") " pod="openshift-image-registry/node-ca-qzm9m"
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754140    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-socket-dir\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.754220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4a459e7e-1c73-4170-b753-22882c81fa6d-agent-certs\") pod \"konnectivity-agent-dlwht\" (UID: \"4a459e7e-1c73-4170-b753-22882c81fa6d\") " pod="kube-system/konnectivity-agent-dlwht"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-sys-fs\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zzbq\" (UniqueName: \"kubernetes.io/projected/3cd98fb6-5344-470a-a91b-c491ba24c89a-kube-api-access-7zzbq\") pod \"iptables-alerter-749nw\" (UID: \"3cd98fb6-5344-470a-a91b-c491ba24c89a\") " pod="openshift-network-operator/iptables-alerter-749nw"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-run-netns\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754332    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-sysctl-conf\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-sys\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-systemd-units\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-system-cni-dir\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-hostroot\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-multus-daemon-config\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-var-lib-openvswitch\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/50a056f9-6c58-4c10-91e6-4c8202e3cc20-ovnkube-config\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-var-lib-kubelet\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754654    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22254a42-55b7-4718-abf3-475178a65fc9-tmp-dir\") pod \"node-resolver-7rg26\" (UID: \"22254a42-55b7-4718-abf3-475178a65fc9\") " pod="openshift-dns/node-resolver-7rg26"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-sysctl-d\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.755076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754703    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-systemd\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754727    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-host\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754740    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-run-ovn-kubernetes\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-cni-binary-copy\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-run-multus-certs\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22254a42-55b7-4718-abf3-475178a65fc9-hosts-file\") pod \"node-resolver-7rg26\" (UID: \"22254a42-55b7-4718-abf3-475178a65fc9\") " pod="openshift-dns/node-resolver-7rg26"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-run-netns\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.754982    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-run-ovn\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-node-log\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-cni-netd\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755112    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/50a056f9-6c58-4c10-91e6-4c8202e3cc20-ovn-node-metrics-cert\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755187    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-run-k8s-cni-cncf-io\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80100b19-7ac7-4382-af81-3f0a97bd7072-kubelet-config\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-device-dir\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.755899 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-cni-binary-copy\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-etc-selinux\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-cnibin\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-device-dir\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-run\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pvkj\" (UniqueName: \"kubernetes.io/projected/5638697b-24e8-4230-857d-7add1666f79a-kube-api-access-5pvkj\") pod \"node-ca-qzm9m\" (UID: \"5638697b-24e8-4230-857d-7add1666f79a\") " pod="openshift-image-registry/node-ca-qzm9m"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755470    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-multus-cni-dir\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-multus-conf-dir\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llx5r\" (UniqueName: \"kubernetes.io/projected/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-kube-api-access-llx5r\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-modprobe-d\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e788dcd7-a1ba-4fab-a0d1-52871fbae098-tmp\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9jb4\" (UniqueName: \"kubernetes.io/projected/e788dcd7-a1ba-4fab-a0d1-52871fbae098-kube-api-access-v9jb4\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755611    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-etc-selinux\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4a459e7e-1c73-4170-b753-22882c81fa6d-konnectivity-ca\") pod \"konnectivity-agent-dlwht\" (UID: \"4a459e7e-1c73-4170-b753-22882c81fa6d\") " pod="kube-system/konnectivity-agent-dlwht"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-registration-dir\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-sysconfig\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.756794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:33.757632 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-etc-openvswitch\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.757632 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.755906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe144550-4e3b-4c9a-8068-4dd0d03bd209-registration-dir\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.757632 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.756256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4a459e7e-1c73-4170-b753-22882c81fa6d-konnectivity-ca\") pod \"konnectivity-agent-dlwht\" (UID: \"4a459e7e-1c73-4170-b753-22882c81fa6d\") " pod="kube-system/konnectivity-agent-dlwht"
Sep 16 17:24:33.759290 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.759261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4a459e7e-1c73-4170-b753-22882c81fa6d-agent-certs\") pod \"konnectivity-agent-dlwht\" (UID: \"4a459e7e-1c73-4170-b753-22882c81fa6d\") " pod="kube-system/konnectivity-agent-dlwht"
Sep 16 17:24:33.768426 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:33.768401    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:33.768426 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:33.768427    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:33.768631 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:33.768440    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2hkhb for pod openshift-network-diagnostics/network-check-target-ppgtg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:33.768631 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:33.768502    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb podName:bde6ac75-b65f-4477-829e-4e37b5c55b19 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:34.268485406 +0000 UTC m=+3.057107919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2hkhb" (UniqueName: "kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb") pod "network-check-target-ppgtg" (UID: "bde6ac75-b65f-4477-829e-4e37b5c55b19") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:33.769935 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.769888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gkr9\" (UniqueName: \"kubernetes.io/projected/fe144550-4e3b-4c9a-8068-4dd0d03bd209-kube-api-access-9gkr9\") pod \"aws-ebs-csi-driver-node-fz5l4\" (UID: \"fe144550-4e3b-4c9a-8068-4dd0d03bd209\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.770575 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.770555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zzbq\" (UniqueName: \"kubernetes.io/projected/3cd98fb6-5344-470a-a91b-c491ba24c89a-kube-api-access-7zzbq\") pod \"iptables-alerter-749nw\" (UID: \"3cd98fb6-5344-470a-a91b-c491ba24c89a\") " pod="openshift-network-operator/iptables-alerter-749nw"
Sep 16 17:24:33.771764 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.771738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgkw9\" (UniqueName: \"kubernetes.io/projected/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-kube-api-access-kgkw9\") pod \"network-metrics-daemon-jhf5t\" (UID: \"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8\") " pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:33.777472 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.777450    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pvkj\" (UniqueName: \"kubernetes.io/projected/5638697b-24e8-4230-857d-7add1666f79a-kube-api-access-5pvkj\") pod \"node-ca-qzm9m\" (UID: \"5638697b-24e8-4230-857d-7add1666f79a\") " pod="openshift-image-registry/node-ca-qzm9m"
Sep 16 17:24:33.777932 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.777896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgk85\" (UniqueName: \"kubernetes.io/projected/1daba9ed-cc2c-4e25-b904-1e2a34af7ce1-kube-api-access-pgk85\") pod \"multus-additional-cni-plugins-nn9tr\" (UID: \"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1\") " pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.856364 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-multus-socket-dir-parent\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.856364 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjpjm\" (UniqueName: \"kubernetes.io/projected/22254a42-55b7-4718-abf3-475178a65fc9-kube-api-access-vjpjm\") pod \"node-resolver-7rg26\" (UID: \"22254a42-55b7-4718-abf3-475178a65fc9\") " pod="openshift-dns/node-resolver-7rg26"
Sep 16 17:24:33.856630 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-run-systemd\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.856630 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-var-lib-cni-bin\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.856630 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856439    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-etc-kubernetes\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.856630 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856499    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-var-lib-cni-bin\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.856630 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856506    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-multus-socket-dir-parent\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.856630 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856502    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-etc-kubernetes\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.856630 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-run-systemd\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.856630 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-lib-modules\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.856902 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-slash\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.856902 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-cni-bin\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.856902 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.856902 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856742    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-slash\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.856902 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856799    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-cni-bin\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.856902 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856828    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.857139 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856928    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-lib-modules\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.857139 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-cni-binary-copy\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.857139 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-var-lib-cni-multus\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.857139 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.856996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-tuned\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.857139 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7c29\" (UniqueName: \"kubernetes.io/projected/50a056f9-6c58-4c10-91e6-4c8202e3cc20-kube-api-access-c7c29\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.857139 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-kubernetes\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.857139 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-run-openvswitch\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.857438 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/50a056f9-6c58-4c10-91e6-4c8202e3cc20-ovnkube-script-lib\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.857438 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-kubelet\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.857438 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-run-netns\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.857438 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-sysctl-conf\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.857438 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-sys\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.857438 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-cni-binary-copy\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.857438 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-systemd-units\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.857438 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-run-openvswitch\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.857438 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857403    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-system-cni-dir\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-var-lib-cni-multus\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-hostroot\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-multus-daemon-config\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-var-lib-openvswitch\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-system-cni-dir\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857534    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/50a056f9-6c58-4c10-91e6-4c8202e3cc20-ovnkube-config\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-var-lib-kubelet\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22254a42-55b7-4718-abf3-475178a65fc9-tmp-dir\") pod \"node-resolver-7rg26\" (UID: \"22254a42-55b7-4718-abf3-475178a65fc9\") " pod="openshift-dns/node-resolver-7rg26"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-sysctl-d\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-systemd\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-host\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-run-ovn-kubernetes\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-run-multus-certs\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22254a42-55b7-4718-abf3-475178a65fc9-hosts-file\") pod \"node-resolver-7rg26\" (UID: \"22254a42-55b7-4718-abf3-475178a65fc9\") " pod="openshift-dns/node-resolver-7rg26"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-run-netns\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857741    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-run-ovn\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-node-log\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.857795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-cni-netd\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/50a056f9-6c58-4c10-91e6-4c8202e3cc20-ovn-node-metrics-cert\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-run-k8s-cni-cncf-io\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80100b19-7ac7-4382-af81-3f0a97bd7072-kubelet-config\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-cnibin\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-run\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-multus-cni-dir\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-multus-conf-dir\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llx5r\" (UniqueName: \"kubernetes.io/projected/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-kube-api-access-llx5r\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.857987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-modprobe-d\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e788dcd7-a1ba-4fab-a0d1-52871fbae098-tmp\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858023    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9jb4\" (UniqueName: \"kubernetes.io/projected/e788dcd7-a1ba-4fab-a0d1-52871fbae098-kube-api-access-v9jb4\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-sysconfig\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858099    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-etc-openvswitch\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-var-lib-kubelet\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858158    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-log-socket\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/50a056f9-6c58-4c10-91e6-4c8202e3cc20-env-overrides\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.858505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858174    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/50a056f9-6c58-4c10-91e6-4c8202e3cc20-ovnkube-script-lib\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.859278 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80100b19-7ac7-4382-af81-3f0a97bd7072-dbus\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:33.859278 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858277    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-cni-netd\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.859278 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858539    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-sysconfig\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.859278 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-run-k8s-cni-cncf-io\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.859278 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80100b19-7ac7-4382-af81-3f0a97bd7072-kubelet-config\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:33.859278 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-cnibin\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.859278 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858736    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-run\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.859278 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.858869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-multus-cni-dir\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.859278 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.859088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-multus-conf-dir\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.859278 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:33.859123    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:33.859780 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.859389    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-modprobe-d\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.859780 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.859447    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-log-socket\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.859780 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.859469    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-sysctl-conf\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.859780 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.859473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-var-lib-openvswitch\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.859780 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.859514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-sys\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.859780 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:33.859521    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret podName:80100b19-7ac7-4382-af81-3f0a97bd7072 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:34.359501868 +0000 UTC m=+3.148124390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret") pod "global-pull-secret-syncer-jxft6" (UID: "80100b19-7ac7-4382-af81-3f0a97bd7072") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:33.859780 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.859527    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-kubernetes\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.859780 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.859557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-etc-openvswitch\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.859780 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.859570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-systemd-units\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.859780 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.859594    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80100b19-7ac7-4382-af81-3f0a97bd7072-dbus\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:33.860521 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.859897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/50a056f9-6c58-4c10-91e6-4c8202e3cc20-env-overrides\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.860521 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860217    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/50a056f9-6c58-4c10-91e6-4c8202e3cc20-ovnkube-config\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.860521 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-kubelet\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.860521 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-run-netns\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.860521 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860315    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-multus-daemon-config\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.860521 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-run-ovn-kubernetes\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.860521 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860418    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-sysctl-d\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.860521 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860508    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22254a42-55b7-4718-abf3-475178a65fc9-tmp-dir\") pod \"node-resolver-7rg26\" (UID: \"22254a42-55b7-4718-abf3-475178a65fc9\") " pod="openshift-dns/node-resolver-7rg26"
Sep 16 17:24:33.860939 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-var-lib-kubelet\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.860939 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860562    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-var-lib-kubelet\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.860939 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22254a42-55b7-4718-abf3-475178a65fc9-hosts-file\") pod \"node-resolver-7rg26\" (UID: \"22254a42-55b7-4718-abf3-475178a65fc9\") " pod="openshift-dns/node-resolver-7rg26"
Sep 16 17:24:33.860939 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860599    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-systemd\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.860939 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860613    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-run-ovn\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.860939 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860619    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-host-run-multus-certs\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.860939 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e788dcd7-a1ba-4fab-a0d1-52871fbae098-host\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.860939 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-host-run-netns\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.860939 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860727    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/50a056f9-6c58-4c10-91e6-4c8202e3cc20-node-log\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.860939 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.860717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-hostroot\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.861440 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.861187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/50a056f9-6c58-4c10-91e6-4c8202e3cc20-ovn-node-metrics-cert\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.861440 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.861251    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e788dcd7-a1ba-4fab-a0d1-52871fbae098-etc-tuned\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.862149 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.862129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e788dcd7-a1ba-4fab-a0d1-52871fbae098-tmp\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.873225 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.873153    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9jb4\" (UniqueName: \"kubernetes.io/projected/e788dcd7-a1ba-4fab-a0d1-52871fbae098-kube-api-access-v9jb4\") pod \"tuned-bvnzw\" (UID: \"e788dcd7-a1ba-4fab-a0d1-52871fbae098\") " pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:33.873619 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.873597    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7c29\" (UniqueName: \"kubernetes.io/projected/50a056f9-6c58-4c10-91e6-4c8202e3cc20-kube-api-access-c7c29\") pod \"ovnkube-node-5kpv5\" (UID: \"50a056f9-6c58-4c10-91e6-4c8202e3cc20\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:33.873842 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.873818    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjpjm\" (UniqueName: \"kubernetes.io/projected/22254a42-55b7-4718-abf3-475178a65fc9-kube-api-access-vjpjm\") pod \"node-resolver-7rg26\" (UID: \"22254a42-55b7-4718-abf3-475178a65fc9\") " pod="openshift-dns/node-resolver-7rg26"
Sep 16 17:24:33.874319 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.874299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llx5r\" (UniqueName: \"kubernetes.io/projected/bdf31547-ab38-4cf0-81d4-c0c02bc3e54e-kube-api-access-llx5r\") pod \"multus-ncvfq\" (UID: \"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e\") " pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.941591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.941556    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-749nw"
Sep 16 17:24:33.950698 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.950674    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dlwht"
Sep 16 17:24:33.961485 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.961459    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nn9tr"
Sep 16 17:24:33.968657 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.968633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4"
Sep 16 17:24:33.976583 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.976548    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qzm9m"
Sep 16 17:24:33.984681 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.984651    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ncvfq"
Sep 16 17:24:33.994635 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:33.994611    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7rg26"
Sep 16 17:24:34.002781 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.002752    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bvnzw"
Sep 16 17:24:34.007808 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.007786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:34.262164 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.262067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs\") pod \"network-metrics-daemon-jhf5t\" (UID: \"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8\") " pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:34.262329 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:34.262233    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:34.262329 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:34.262304    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs podName:fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:35.262282832 +0000 UTC m=+4.050905370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs") pod "network-metrics-daemon-jhf5t" (UID: "fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:34.363344 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.363290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:34.363344 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.363350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hkhb\" (UniqueName: \"kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb\") pod \"network-check-target-ppgtg\" (UID: \"bde6ac75-b65f-4477-829e-4e37b5c55b19\") " pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:34.363582 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:34.363472    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:34.363582 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:34.363506    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:34.363582 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:34.363525    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:34.363582 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:34.363540    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2hkhb for pod openshift-network-diagnostics/network-check-target-ppgtg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:34.363582 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:34.363561    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret podName:80100b19-7ac7-4382-af81-3f0a97bd7072 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:35.363540303 +0000 UTC m=+4.152162830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret") pod "global-pull-secret-syncer-jxft6" (UID: "80100b19-7ac7-4382-af81-3f0a97bd7072") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:34.363816 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:34.363597    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb podName:bde6ac75-b65f-4477-829e-4e37b5c55b19 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:35.363580683 +0000 UTC m=+4.152203219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hkhb" (UniqueName: "kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb") pod "network-check-target-ppgtg" (UID: "bde6ac75-b65f-4477-829e-4e37b5c55b19") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:34.727068 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:34.727041    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a056f9_6c58_4c10_91e6_4c8202e3cc20.slice/crio-2f7ef985485a9c1546bec95022f5ee6ce70a853253681ebcbcbae24726cc8546 WatchSource:0}: Error finding container 2f7ef985485a9c1546bec95022f5ee6ce70a853253681ebcbcbae24726cc8546: Status 404 returned error can't find the container with id 2f7ef985485a9c1546bec95022f5ee6ce70a853253681ebcbcbae24726cc8546
Sep 16 17:24:34.728210 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:34.728185    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd98fb6_5344_470a_a91b_c491ba24c89a.slice/crio-64db5c61d6b773c08532149cb0ae5e02d314944bb1b69c30bbcf5ede4bf7f835 WatchSource:0}: Error finding container 64db5c61d6b773c08532149cb0ae5e02d314944bb1b69c30bbcf5ede4bf7f835: Status 404 returned error can't find the container with id 64db5c61d6b773c08532149cb0ae5e02d314944bb1b69c30bbcf5ede4bf7f835
Sep 16 17:24:34.730038 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:34.729831    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf31547_ab38_4cf0_81d4_c0c02bc3e54e.slice/crio-ed222d858667685a86a7fb43f4678deb9de8697f6bf74bc82ff0bcb9c271b706 WatchSource:0}: Error finding container ed222d858667685a86a7fb43f4678deb9de8697f6bf74bc82ff0bcb9c271b706: Status 404 returned error can't find the container with id ed222d858667685a86a7fb43f4678deb9de8697f6bf74bc82ff0bcb9c271b706
Sep 16 17:24:34.732548 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:34.732392    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22254a42_55b7_4718_abf3_475178a65fc9.slice/crio-eab16b65e304be2ed0827379c5f5d49876a290998529f8be765877257877c43b WatchSource:0}: Error finding container eab16b65e304be2ed0827379c5f5d49876a290998529f8be765877257877c43b: Status 404 returned error can't find the container with id eab16b65e304be2ed0827379c5f5d49876a290998529f8be765877257877c43b
Sep 16 17:24:34.734165 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:34.734072    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5638697b_24e8_4230_857d_7add1666f79a.slice/crio-d6c5a7c547907df5351e496257789662f988ee1db5a7f3c6f1b258b3aa155444 WatchSource:0}: Error finding container d6c5a7c547907df5351e496257789662f988ee1db5a7f3c6f1b258b3aa155444: Status 404 returned error can't find the container with id d6c5a7c547907df5351e496257789662f988ee1db5a7f3c6f1b258b3aa155444
Sep 16 17:24:34.735509 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:34.735481    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a459e7e_1c73_4170_b753_22882c81fa6d.slice/crio-917df567da9afe59222aa570e7cf0c7b15319e026c26ac6104c0192fbf6540c7 WatchSource:0}: Error finding container 917df567da9afe59222aa570e7cf0c7b15319e026c26ac6104c0192fbf6540c7: Status 404 returned error can't find the container with id 917df567da9afe59222aa570e7cf0c7b15319e026c26ac6104c0192fbf6540c7
Sep 16 17:24:34.737491 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:34.736590    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode788dcd7_a1ba_4fab_a0d1_52871fbae098.slice/crio-2d5e1d3ceaaa9b3bce76c0cad6fab9495569a043e2fdea800fbc7f2402ced658 WatchSource:0}: Error finding container 2d5e1d3ceaaa9b3bce76c0cad6fab9495569a043e2fdea800fbc7f2402ced658: Status 404 returned error can't find the container with id 2d5e1d3ceaaa9b3bce76c0cad6fab9495569a043e2fdea800fbc7f2402ced658
Sep 16 17:24:34.737713 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:34.737663    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe144550_4e3b_4c9a_8068_4dd0d03bd209.slice/crio-8b883a004cfe51441935813db598fc43f0543d9cc80d93c561be6921a650a94c WatchSource:0}: Error finding container 8b883a004cfe51441935813db598fc43f0543d9cc80d93c561be6921a650a94c: Status 404 returned error can't find the container with id 8b883a004cfe51441935813db598fc43f0543d9cc80d93c561be6921a650a94c
Sep 16 17:24:34.740659 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:24:34.740583    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1daba9ed_cc2c_4e25_b904_1e2a34af7ce1.slice/crio-0fc7ee9d77f5a912c57bfd83003d96e294f0add990426ef4e77c81267761682c WatchSource:0}: Error finding container 0fc7ee9d77f5a912c57bfd83003d96e294f0add990426ef4e77c81267761682c: Status 404 returned error can't find the container with id 0fc7ee9d77f5a912c57bfd83003d96e294f0add990426ef4e77c81267761682c
Sep 16 17:24:34.741166 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.741133    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 17:19:33 +0000 UTC" deadline="2028-04-14 11:45:23.803429115 +0000 UTC"
Sep 16 17:24:34.741166 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.741166    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13818h20m49.062267744s"
Sep 16 17:24:34.760829 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.760797    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:34.760953 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.760850    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:34.761006 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:34.760967    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:24:34.761077 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:34.761058    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:24:34.772770 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.772737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4" event={"ID":"fe144550-4e3b-4c9a-8068-4dd0d03bd209","Type":"ContainerStarted","Data":"8b883a004cfe51441935813db598fc43f0543d9cc80d93c561be6921a650a94c"}
Sep 16 17:24:34.776689 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.776657    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dlwht" event={"ID":"4a459e7e-1c73-4170-b753-22882c81fa6d","Type":"ContainerStarted","Data":"917df567da9afe59222aa570e7cf0c7b15319e026c26ac6104c0192fbf6540c7"}
Sep 16 17:24:34.781435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.781407    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ncvfq" event={"ID":"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e","Type":"ContainerStarted","Data":"ed222d858667685a86a7fb43f4678deb9de8697f6bf74bc82ff0bcb9c271b706"}
Sep 16 17:24:34.783508 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.783484    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-749nw" event={"ID":"3cd98fb6-5344-470a-a91b-c491ba24c89a","Type":"ContainerStarted","Data":"64db5c61d6b773c08532149cb0ae5e02d314944bb1b69c30bbcf5ede4bf7f835"}
Sep 16 17:24:34.784840 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.784817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5" event={"ID":"50a056f9-6c58-4c10-91e6-4c8202e3cc20","Type":"ContainerStarted","Data":"2f7ef985485a9c1546bec95022f5ee6ce70a853253681ebcbcbae24726cc8546"}
Sep 16 17:24:34.787091 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.787064    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nn9tr" event={"ID":"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1","Type":"ContainerStarted","Data":"0fc7ee9d77f5a912c57bfd83003d96e294f0add990426ef4e77c81267761682c"}
Sep 16 17:24:34.789237 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.789211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bvnzw" event={"ID":"e788dcd7-a1ba-4fab-a0d1-52871fbae098","Type":"ContainerStarted","Data":"2d5e1d3ceaaa9b3bce76c0cad6fab9495569a043e2fdea800fbc7f2402ced658"}
Sep 16 17:24:34.793854 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.793823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qzm9m" event={"ID":"5638697b-24e8-4230-857d-7add1666f79a","Type":"ContainerStarted","Data":"d6c5a7c547907df5351e496257789662f988ee1db5a7f3c6f1b258b3aa155444"}
Sep 16 17:24:34.798273 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:34.798254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7rg26" event={"ID":"22254a42-55b7-4718-abf3-475178a65fc9","Type":"ContainerStarted","Data":"eab16b65e304be2ed0827379c5f5d49876a290998529f8be765877257877c43b"}
Sep 16 17:24:35.272050 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:35.271928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs\") pod \"network-metrics-daemon-jhf5t\" (UID: \"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8\") " pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:35.272241 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:35.272223    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:35.272324 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:35.272308    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs podName:fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:37.272288509 +0000 UTC m=+6.060911047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs") pod "network-metrics-daemon-jhf5t" (UID: "fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:35.372756 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:35.372707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:35.372956 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:35.372772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hkhb\" (UniqueName: \"kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb\") pod \"network-check-target-ppgtg\" (UID: \"bde6ac75-b65f-4477-829e-4e37b5c55b19\") " pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:35.373123 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:35.372969    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:35.373123 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:35.372990    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:35.373123 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:35.373003    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2hkhb for pod openshift-network-diagnostics/network-check-target-ppgtg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:35.373123 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:35.373064    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb podName:bde6ac75-b65f-4477-829e-4e37b5c55b19 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:37.373045238 +0000 UTC m=+6.161667765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hkhb" (UniqueName: "kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb") pod "network-check-target-ppgtg" (UID: "bde6ac75-b65f-4477-829e-4e37b5c55b19") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:35.383369 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:35.373500    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:35.383369 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:35.373551    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret podName:80100b19-7ac7-4382-af81-3f0a97bd7072 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:37.373536093 +0000 UTC m=+6.162158609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret") pod "global-pull-secret-syncer-jxft6" (UID: "80100b19-7ac7-4382-af81-3f0a97bd7072") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:35.741954 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:35.741776    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 17:19:33 +0000 UTC" deadline="2028-03-29 10:24:04.018545975 +0000 UTC"
Sep 16 17:24:35.741954 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:35.741820    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13432h59m28.276730788s"
Sep 16 17:24:35.762587 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:35.761583    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:35.762587 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:35.761722    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:24:35.828383 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:35.828333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" event={"ID":"c40b89e27f34846eb7c4becaeceb3ee5","Type":"ContainerStarted","Data":"7609a68741cf4bce02999d66600a41ebc4f9cd1ae5b8f41834f066d50337b8ef"}
Sep 16 17:24:35.845262 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:35.845219    2581 generic.go:356] "Generic (PLEG): container finished" podID="4a009e1b07a5529d711c19429f741484" containerID="c8b3e84dea95fddd85ba93c6dc3e100d05bafba56979d3a5f8258f325642b856" exitCode=0
Sep 16 17:24:35.845435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:35.845286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" event={"ID":"4a009e1b07a5529d711c19429f741484","Type":"ContainerDied","Data":"c8b3e84dea95fddd85ba93c6dc3e100d05bafba56979d3a5f8258f325642b856"}
Sep 16 17:24:35.850874 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:35.850789    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" podStartSLOduration=2.85076897 podStartE2EDuration="2.85076897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:35.850008761 +0000 UTC m=+4.638631297" watchObservedRunningTime="2026-09-16 17:24:35.85076897 +0000 UTC m=+4.639391508"
Sep 16 17:24:36.761361 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:36.761101    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:36.761869 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:36.761515    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:24:36.762015 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:36.761937    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:36.762087 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:36.762059    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:24:36.867258 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:36.867222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" event={"ID":"4a009e1b07a5529d711c19429f741484","Type":"ContainerStarted","Data":"b42dd64798095cd400f3efb9de48b85000bdf37e4e01885788b23afd3532f1d1"}
Sep 16 17:24:37.291822 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:37.291778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs\") pod \"network-metrics-daemon-jhf5t\" (UID: \"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8\") " pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:37.292026 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:37.291953    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:37.292026 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:37.292018    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs podName:fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:41.291997277 +0000 UTC m=+10.080619804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs") pod "network-metrics-daemon-jhf5t" (UID: "fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:37.393145 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:37.393105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:37.393311 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:37.393160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hkhb\" (UniqueName: \"kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb\") pod \"network-check-target-ppgtg\" (UID: \"bde6ac75-b65f-4477-829e-4e37b5c55b19\") " pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:37.393392 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:37.393344    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:37.393392 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:37.393365    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:37.393392 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:37.393380    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2hkhb for pod openshift-network-diagnostics/network-check-target-ppgtg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:37.393527 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:37.393437    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb podName:bde6ac75-b65f-4477-829e-4e37b5c55b19 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:41.39341885 +0000 UTC m=+10.182041376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hkhb" (UniqueName: "kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb") pod "network-check-target-ppgtg" (UID: "bde6ac75-b65f-4477-829e-4e37b5c55b19") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:37.393835 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:37.393815    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:37.393939 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:37.393871    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret podName:80100b19-7ac7-4382-af81-3f0a97bd7072 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:41.393857345 +0000 UTC m=+10.182479860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret") pod "global-pull-secret-syncer-jxft6" (UID: "80100b19-7ac7-4382-af81-3f0a97bd7072") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:37.762984 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:37.762884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:37.763503 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:37.762999    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:24:38.760884 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:38.760851    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:38.761074 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:38.761003    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:24:38.761533 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:38.761368    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:38.761533 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:38.761460    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:24:39.765418 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:39.765386    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:39.765883 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:39.765505    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:24:40.761429 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:40.761374    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:40.761429 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:40.761425    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:40.761724 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:40.761532    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:24:40.761724 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:40.761669    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:24:41.325561 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:41.325525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs\") pod \"network-metrics-daemon-jhf5t\" (UID: \"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8\") " pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:41.326031 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:41.325651    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:41.326031 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:41.325712    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs podName:fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:49.325692918 +0000 UTC m=+18.114315434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs") pod "network-metrics-daemon-jhf5t" (UID: "fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:41.426160 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:41.426123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:41.426160 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:41.426167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hkhb\" (UniqueName: \"kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb\") pod \"network-check-target-ppgtg\" (UID: \"bde6ac75-b65f-4477-829e-4e37b5c55b19\") " pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:41.426384 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:41.426276    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:41.426384 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:41.426295    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:41.426384 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:41.426310    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:41.426384 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:41.426322    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2hkhb for pod openshift-network-diagnostics/network-check-target-ppgtg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:41.426384 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:41.426342    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret podName:80100b19-7ac7-4382-af81-3f0a97bd7072 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:49.42632773 +0000 UTC m=+18.214950242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret") pod "global-pull-secret-syncer-jxft6" (UID: "80100b19-7ac7-4382-af81-3f0a97bd7072") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:41.426384 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:41.426369    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb podName:bde6ac75-b65f-4477-829e-4e37b5c55b19 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:49.426354331 +0000 UTC m=+18.214976868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hkhb" (UniqueName: "kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb") pod "network-check-target-ppgtg" (UID: "bde6ac75-b65f-4477-829e-4e37b5c55b19") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:41.761552 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:41.761464    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:41.761728 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:41.761589    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:24:42.761714 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:42.761673    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:42.762161 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:42.761777    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:24:42.762161 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:42.762068    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:42.762161 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:42.762127    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:24:43.761503 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:43.761081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:43.761503 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:43.761219    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:24:44.760843 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:44.760805    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:44.761223 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:44.760806    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:44.761223 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:44.760932    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:24:44.761223 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:44.760982    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:24:45.761286 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:45.761205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:45.761707 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:45.761342    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:24:46.761480 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:46.761446    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:46.761943 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:46.761452    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:46.761943 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:46.761555    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:24:46.761943 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:46.761686    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:24:47.760979 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:47.760944    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:47.761130 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:47.761076    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:24:48.760840 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:48.760795    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:48.761413 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:48.760950    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:24:48.761413 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:48.760804    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:48.761413 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:48.761253    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:24:49.387041 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:49.386972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs\") pod \"network-metrics-daemon-jhf5t\" (UID: \"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8\") " pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:49.387196 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:49.387171    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:49.387279 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:49.387267    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs podName:fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:05.387242039 +0000 UTC m=+34.175864568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs") pod "network-metrics-daemon-jhf5t" (UID: "fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:49.487713 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:49.487682    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:49.487713 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:49.487725    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hkhb\" (UniqueName: \"kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb\") pod \"network-check-target-ppgtg\" (UID: \"bde6ac75-b65f-4477-829e-4e37b5c55b19\") " pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:49.487922 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:49.487825    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:49.487922 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:49.487832    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:49.487922 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:49.487851    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:49.487922 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:49.487859    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2hkhb for pod openshift-network-diagnostics/network-check-target-ppgtg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:49.487922 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:49.487900    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret podName:80100b19-7ac7-4382-af81-3f0a97bd7072 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:05.487874094 +0000 UTC m=+34.276496606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret") pod "global-pull-secret-syncer-jxft6" (UID: "80100b19-7ac7-4382-af81-3f0a97bd7072") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:49.487922 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:49.487921    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb podName:bde6ac75-b65f-4477-829e-4e37b5c55b19 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:05.487910887 +0000 UTC m=+34.276533401 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hkhb" (UniqueName: "kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb") pod "network-check-target-ppgtg" (UID: "bde6ac75-b65f-4477-829e-4e37b5c55b19") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:49.763826 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:49.763754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:49.764234 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:49.763881    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:24:50.760706 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:50.760669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:50.760870 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:50.760800    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:24:50.761226 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:50.761198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:50.761349 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:50.761307    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:24:51.764223 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:51.764196    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:51.764663 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:51.764293    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:24:52.762000 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:52.761347    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:52.762000 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:52.761475    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:24:52.762000 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:52.761831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:52.762000 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:52.761963    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:24:52.913479 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:52.913040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7rg26" event={"ID":"22254a42-55b7-4718-abf3-475178a65fc9","Type":"ContainerStarted","Data":"1b126a7f6b669ee63e456eed707ba46e58c271cd5ed36b5b26f27ed523d00b86"}
Sep 16 17:24:52.917727 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:52.917700    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dlwht" event={"ID":"4a459e7e-1c73-4170-b753-22882c81fa6d","Type":"ContainerStarted","Data":"4453a033d5f348c4c866bb8556fab57f9b95abada14c3f4990631e661fe05700"}
Sep 16 17:24:52.919511 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:52.919483    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ncvfq" event={"ID":"bdf31547-ab38-4cf0-81d4-c0c02bc3e54e","Type":"ContainerStarted","Data":"35e720f29f90a389c1ba277b81fa55dbba455ff318efb2c7cb30d88204294460"}
Sep 16 17:24:52.921909 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:52.921857    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nn9tr" event={"ID":"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1","Type":"ContainerStarted","Data":"9f5278882289cb50628f937d08e8b39ce5a76234251e1c46c6bc2776a39550c7"}
Sep 16 17:24:52.924074 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:52.924046    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bvnzw" event={"ID":"e788dcd7-a1ba-4fab-a0d1-52871fbae098","Type":"ContainerStarted","Data":"789bb871103344026dc1b05013e1ced9b203bd2f07966c09a90b7ac5540f74ab"}
Sep 16 17:24:52.930924 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:52.930618    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7rg26" podStartSLOduration=2.978349567 podStartE2EDuration="20.930601448s" totalImagesPullingTime="17.952251881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:52.930596545 +0000 UTC m=+21.719219080" watchObservedRunningTime="2026-09-16 17:24:52.930601448 +0000 UTC m=+21.719223982"
Sep 16 17:24:52.931035 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:52.930938    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" podStartSLOduration=19.930931742 podStartE2EDuration="19.930931742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:36.890052068 +0000 UTC m=+5.678674606" watchObservedRunningTime="2026-09-16 17:24:52.930931742 +0000 UTC m=+21.719554460"
Sep 16 17:24:52.970706 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:52.970572    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ncvfq" podStartSLOduration=2.915813177 podStartE2EDuration="20.9705571s" totalImagesPullingTime="18.054743923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:52.970084113 +0000 UTC m=+21.758706744" watchObservedRunningTime="2026-09-16 17:24:52.9705571 +0000 UTC m=+21.759179652"
Sep 16 17:24:52.990731 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:52.990523    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bvnzw" podStartSLOduration=3.022281336 podStartE2EDuration="20.990503788s" totalImagesPullingTime="17.968222452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:52.990481386 +0000 UTC m=+21.779103919" watchObservedRunningTime="2026-09-16 17:24:52.990503788 +0000 UTC m=+21.779126319"
Sep 16 17:24:53.761131 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:53.760937    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:53.761461 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:53.761233    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:24:53.927381 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:53.927343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4" event={"ID":"fe144550-4e3b-4c9a-8068-4dd0d03bd209","Type":"ContainerStarted","Data":"9fffd5c6c454ff909345cfa7299a8af2bb68caaef20768c3d3c9611828244080"}
Sep 16 17:24:53.930144 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:53.930120    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5" event={"ID":"50a056f9-6c58-4c10-91e6-4c8202e3cc20","Type":"ContainerStarted","Data":"df3bd939943aa34b7ee3c315e23abb4500831a2b7a612d49b5298a038575953c"}
Sep 16 17:24:53.930282 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:53.930191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5" event={"ID":"50a056f9-6c58-4c10-91e6-4c8202e3cc20","Type":"ContainerStarted","Data":"c7ce206a9a8a79bde9c278ca98c80c832a047768a9d5061d71fcc292f60ae956"}
Sep 16 17:24:53.930282 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:53.930236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5" event={"ID":"50a056f9-6c58-4c10-91e6-4c8202e3cc20","Type":"ContainerStarted","Data":"6fc73300cd040dbdb4c41095cabe2e41ec75916266c5da9c5cea4e4c292a120f"}
Sep 16 17:24:53.930282 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:53.930268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5" event={"ID":"50a056f9-6c58-4c10-91e6-4c8202e3cc20","Type":"ContainerStarted","Data":"f1d8a183245fe3674d9350b076002c47798623ddc0364e9c5836104bcdfd47a7"}
Sep 16 17:24:53.930431 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:53.930295    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5" event={"ID":"50a056f9-6c58-4c10-91e6-4c8202e3cc20","Type":"ContainerStarted","Data":"02a0f48df510b773ca6b524fd3b77bc7dd2405fe914ed0f1e63a5f834041ed2c"}
Sep 16 17:24:53.930431 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:53.930335    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5" event={"ID":"50a056f9-6c58-4c10-91e6-4c8202e3cc20","Type":"ContainerStarted","Data":"674fc353c510f3ce35ae65b8914a3faef8d5b4aaf5b780aee99970554dbe21a4"}
Sep 16 17:24:53.931469 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:53.931447    2581 generic.go:356] "Generic (PLEG): container finished" podID="1daba9ed-cc2c-4e25-b904-1e2a34af7ce1" containerID="9f5278882289cb50628f937d08e8b39ce5a76234251e1c46c6bc2776a39550c7" exitCode=0
Sep 16 17:24:53.931560 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:53.931505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nn9tr" event={"ID":"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1","Type":"ContainerDied","Data":"9f5278882289cb50628f937d08e8b39ce5a76234251e1c46c6bc2776a39550c7"}
Sep 16 17:24:53.933395 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:53.933367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qzm9m" event={"ID":"5638697b-24e8-4230-857d-7add1666f79a","Type":"ContainerStarted","Data":"f82ea9c47c07f544d783a859f5f158ed855395d2fb9c3f6e8a196b225c999821"}
Sep 16 17:24:53.955691 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:53.955643    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dlwht" podStartSLOduration=7.809649612 podStartE2EDuration="21.95562668s" totalImagesPullingTime="14.145977068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:53.013532312 +0000 UTC m=+21.802154846" watchObservedRunningTime="2026-09-16 17:24:53.95562668 +0000 UTC m=+22.744249213"
Sep 16 17:24:54.250546 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:54.250517    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 17:24:54.712682 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:54.712564    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T17:24:54.250535163Z","UUID":"48baadab-8e53-42cd-86bf-9e7d20cecf53","Handler":null,"Name":"","Endpoint":""}
Sep 16 17:24:54.714414 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:54.714385    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 17:24:54.714552 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:54.714429    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 17:24:54.761418 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:54.761379    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:54.761585 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:54.761500    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:24:54.761839 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:54.761806    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:54.761968 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:54.761938    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:24:54.937870 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:54.937827    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4" event={"ID":"fe144550-4e3b-4c9a-8068-4dd0d03bd209","Type":"ContainerStarted","Data":"28f454aad63b9cad81859434669f608163d9e5ae86900439babd09edb856449b"}
Sep 16 17:24:54.939955 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:54.939918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-749nw" event={"ID":"3cd98fb6-5344-470a-a91b-c491ba24c89a","Type":"ContainerStarted","Data":"7359f411f7f228590d2dbe80d28e07e86bbc9a9d1953f578cdce69f923e4eb16"}
Sep 16 17:24:54.958424 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:54.958370    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-749nw" podStartSLOduration=4.9790988 podStartE2EDuration="22.958354553s" totalImagesPullingTime="17.979255753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:54.958337074 +0000 UTC m=+23.746959607" watchObservedRunningTime="2026-09-16 17:24:54.958354553 +0000 UTC m=+23.746977086"
Sep 16 17:24:54.959040 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:54.959009    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qzm9m" podStartSLOduration=4.996730386 podStartE2EDuration="22.959000117s" totalImagesPullingTime="17.962269731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:53.972643615 +0000 UTC m=+22.761266146" watchObservedRunningTime="2026-09-16 17:24:54.959000117 +0000 UTC m=+23.747622656"
Sep 16 17:24:55.760964 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:55.760932    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:55.761143 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:55.761040    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:24:55.944527 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:55.944427    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4" event={"ID":"fe144550-4e3b-4c9a-8068-4dd0d03bd209","Type":"ContainerStarted","Data":"7efd8ae87fce92f6aaef98ee429ac01ef2d1ea74f4efd2e893934468787ea59e"}
Sep 16 17:24:55.948193 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:55.948161    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5" event={"ID":"50a056f9-6c58-4c10-91e6-4c8202e3cc20","Type":"ContainerStarted","Data":"665c247e0b4e5c8a406d68f20db52e5d682c766e569d8c6b84c2fb0250755c76"}
Sep 16 17:24:55.970402 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:55.970351    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fz5l4" podStartSLOduration=3.811774408 podStartE2EDuration="23.970333155s" totalImagesPullingTime="20.158558747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:55.970022379 +0000 UTC m=+24.758645118" watchObservedRunningTime="2026-09-16 17:24:55.970333155 +0000 UTC m=+24.758955670"
Sep 16 17:24:56.157607 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:56.157574    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dlwht"
Sep 16 17:24:56.158226 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:56.158206    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dlwht"
Sep 16 17:24:56.761076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:56.761045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:56.761282 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:56.761160    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:24:56.761282 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:56.761239    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:56.761404 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:56.761343    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:24:57.284861 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:57.284826    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dlwht"
Sep 16 17:24:57.285561 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:57.285544    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dlwht"
Sep 16 17:24:57.761667 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:57.761632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:57.762341 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:57.762319    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:24:58.761487 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:58.761269    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:24:58.762060 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:58.761352    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:24:58.762060 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:58.761508    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:24:58.762060 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:58.761596    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:24:58.958466 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:58.958434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5" event={"ID":"50a056f9-6c58-4c10-91e6-4c8202e3cc20","Type":"ContainerStarted","Data":"84b7f4d85c6f133cea418745b16f02a5fbde439b6653b88f44b4703c4b3a0f07"}
Sep 16 17:24:58.958671 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:58.958651    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:58.960534 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:58.960510    2581 generic.go:356] "Generic (PLEG): container finished" podID="1daba9ed-cc2c-4e25-b904-1e2a34af7ce1" containerID="6c1db197732e6ecce80812777970a369dd12e1baeccc1c2c3d8b7ba306333a91" exitCode=0
Sep 16 17:24:58.960651 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:58.960588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nn9tr" event={"ID":"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1","Type":"ContainerDied","Data":"6c1db197732e6ecce80812777970a369dd12e1baeccc1c2c3d8b7ba306333a91"}
Sep 16 17:24:58.974744 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:58.974723    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:59.069941 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:59.069868    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5" podStartSLOduration=8.938008842 podStartE2EDuration="27.069853135s" totalImagesPullingTime="18.131844293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:59.007938353 +0000 UTC m=+27.796560887" watchObservedRunningTime="2026-09-16 17:24:59.069853135 +0000 UTC m=+27.858475686"
Sep 16 17:24:59.760769 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:59.760736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:24:59.761041 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:24:59.760849    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:24:59.963493 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:59.963468    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:59.964350 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:59.963551    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:24:59.986767 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:24:59.986742    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:25:00.280002 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:00.279812    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jhf5t"]
Sep 16 17:25:00.280182 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:00.280120    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:25:00.280236 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:00.280216    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:25:00.284725 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:00.284692    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jxft6"]
Sep 16 17:25:00.284843 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:00.284786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:25:00.284882 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:00.284855    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:25:00.332095 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:00.331982    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ppgtg"]
Sep 16 17:25:00.332293 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:00.332117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:25:00.332293 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:00.332208    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:25:00.966867 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:00.966832    2581 generic.go:356] "Generic (PLEG): container finished" podID="1daba9ed-cc2c-4e25-b904-1e2a34af7ce1" containerID="266fe685b41064f108ce403101190cb6611e2c7c702a582134a8c7aaf9e90588" exitCode=0
Sep 16 17:25:00.967260 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:00.966924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nn9tr" event={"ID":"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1","Type":"ContainerDied","Data":"266fe685b41064f108ce403101190cb6611e2c7c702a582134a8c7aaf9e90588"}
Sep 16 17:25:01.762416 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:01.762386    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:25:01.762416 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:01.762409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:25:01.762647 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:01.762496    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:25:01.762952 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:01.762924    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:25:02.760763 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:02.760709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:25:02.761203 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:02.760832    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:25:02.974676 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:02.974597    2581 generic.go:356] "Generic (PLEG): container finished" podID="1daba9ed-cc2c-4e25-b904-1e2a34af7ce1" containerID="50c6c7b0846746abf9f3df58e97c0a6a446a4ffe59567b263097b91601815509" exitCode=0
Sep 16 17:25:02.974794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:02.974671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nn9tr" event={"ID":"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1","Type":"ContainerDied","Data":"50c6c7b0846746abf9f3df58e97c0a6a446a4ffe59567b263097b91601815509"}
Sep 16 17:25:03.761449 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:03.761409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:25:03.761843 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:03.761516    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:25:03.761843 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:03.761547    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:25:03.761843 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:03.761596    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:25:04.761203 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:04.761114    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:25:04.761368 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:04.761246    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:25:05.399110 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:05.399061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs\") pod \"network-metrics-daemon-jhf5t\" (UID: \"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8\") " pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:25:05.399675 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:05.399205    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:25:05.399675 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:05.399278    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs podName:fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:37.39925671 +0000 UTC m=+66.187879221 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs") pod "network-metrics-daemon-jhf5t" (UID: "fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:25:05.499643 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:05.499610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:25:05.499643 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:05.499649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hkhb\" (UniqueName: \"kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb\") pod \"network-check-target-ppgtg\" (UID: \"bde6ac75-b65f-4477-829e-4e37b5c55b19\") " pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:25:05.499880 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:05.499787    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:25:05.499880 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:05.499787    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:25:05.499880 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:05.499802    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:25:05.499880 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:05.499816    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2hkhb for pod openshift-network-diagnostics/network-check-target-ppgtg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:25:05.499880 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:05.499873    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret podName:80100b19-7ac7-4382-af81-3f0a97bd7072 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:37.499851866 +0000 UTC m=+66.288474380 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret") pod "global-pull-secret-syncer-jxft6" (UID: "80100b19-7ac7-4382-af81-3f0a97bd7072") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:25:05.500127 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:05.499918    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb podName:bde6ac75-b65f-4477-829e-4e37b5c55b19 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:37.499907775 +0000 UTC m=+66.288530300 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2hkhb" (UniqueName: "kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb") pod "network-check-target-ppgtg" (UID: "bde6ac75-b65f-4477-829e-4e37b5c55b19") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:25:05.761677 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:05.761641    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:25:05.761873 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:05.761778    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ppgtg" podUID="bde6ac75-b65f-4477-829e-4e37b5c55b19"
Sep 16 17:25:05.762169 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:05.762151    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:25:05.762277 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:05.762244    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jxft6" podUID="80100b19-7ac7-4382-af81-3f0a97bd7072"
Sep 16 17:25:06.551805 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.551776    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeReady"
Sep 16 17:25:06.552244 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.552002    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 17:25:06.667110 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.667072    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b89f8fc75-6zrbw"]
Sep 16 17:25:06.671360 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.671330    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.691450 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:06.691421    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"image-registry-6b89f8fc75-6zrbw\" is forbidden: User \"system:node:ip-10-0-128-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-128-31.ec2.internal' and this object" podUID="d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a" pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.693091 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:06.693056    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-tls\" is forbidden: User \"system:node:ip-10-0-128-31.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-128-31.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret"
Sep 16 17:25:06.693829 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:06.693807    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"registry-dockercfg-gnfmt\" is forbidden: User \"system:node:ip-10-0-128-31.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-128-31.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gnfmt\"" type="*v1.Secret"
Sep 16 17:25:06.693913 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:06.693843    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"installation-pull-secrets\" is forbidden: User \"system:node:ip-10-0-128-31.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-128-31.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret"
Sep 16 17:25:06.694259 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:06.694219    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"trusted-ca\" is forbidden: User \"system:node:ip-10-0-128-31.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-128-31.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap"
Sep 16 17:25:06.694372 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:06.694280    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-private-configuration\" is forbidden: User \"system:node:ip-10-0-128-31.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-128-31.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret"
Sep 16 17:25:06.760730 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.760696    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:25:06.769783 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.769759    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vhb2d\""
Sep 16 17:25:06.782098 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.782068    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 17:25:06.795320 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.795299    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-klh2p"]
Sep 16 17:25:06.798260 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.798237    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zc9kr"]
Sep 16 17:25:06.798408 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.798388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:06.801002 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.800982    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zc9kr"
Sep 16 17:25:06.807975 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.807948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-image-registry-private-configuration\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.808068 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.807990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-certificates\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.808068 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.808034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-ca-trust-extracted\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.808068 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.808061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-trusted-ca\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.808200 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.808097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzwgn\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-kube-api-access-nzwgn\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.808200 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.808168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-bound-sa-token\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.808284 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.808205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.808284 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.808240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-installation-pull-secrets\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.811505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.811046    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 17:25:06.811505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.811076    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 17:25:06.811505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.811349    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 17:25:06.819665 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.819642    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 17:25:06.819770 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.819665    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwbrz\""
Sep 16 17:25:06.820611 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.820592    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jnc4m\""
Sep 16 17:25:06.835827 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.835803    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 17:25:06.843233 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.843210    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b89f8fc75-6zrbw"]
Sep 16 17:25:06.888640 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.888603    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-klh2p"]
Sep 16 17:25:06.909609 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.909584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.909781 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.909619    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:06.909781 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.909647    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-installation-pull-secrets\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.909781 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.909678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t99nh\" (UniqueName: \"kubernetes.io/projected/a9f7a40a-f3c9-4e73-940e-666151905ce4-kube-api-access-t99nh\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:06.909781 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.909762    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swqj5\" (UniqueName: \"kubernetes.io/projected/d9947ef2-8d1a-4b13-b818-b5b2b7d95760-kube-api-access-swqj5\") pod \"ingress-canary-zc9kr\" (UID: \"d9947ef2-8d1a-4b13-b818-b5b2b7d95760\") " pod="openshift-ingress-canary/ingress-canary-zc9kr"
Sep 16 17:25:06.909991 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.909949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-image-registry-private-configuration\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.909991 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.909974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-certificates\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.910064 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.910011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-ca-trust-extracted\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.910064 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.910038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-trusted-ca\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.910064 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.910061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzwgn\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-kube-api-access-nzwgn\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.910201 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.910097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9947ef2-8d1a-4b13-b818-b5b2b7d95760-cert\") pod \"ingress-canary-zc9kr\" (UID: \"d9947ef2-8d1a-4b13-b818-b5b2b7d95760\") " pod="openshift-ingress-canary/ingress-canary-zc9kr"
Sep 16 17:25:06.910201 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.910136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-bound-sa-token\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.910349 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.910297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f7a40a-f3c9-4e73-940e-666151905ce4-config-volume\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:06.910349 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.910328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f7a40a-f3c9-4e73-940e-666151905ce4-tmp-dir\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:06.910636 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.910621    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-certificates\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.910743 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.910727    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-ca-trust-extracted\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.921580 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.921541    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zc9kr"]
Sep 16 17:25:06.946086 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.946060    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-bound-sa-token\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:06.946236 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:06.946220    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzwgn\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-kube-api-access-nzwgn\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:07.011059 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.011020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9947ef2-8d1a-4b13-b818-b5b2b7d95760-cert\") pod \"ingress-canary-zc9kr\" (UID: \"d9947ef2-8d1a-4b13-b818-b5b2b7d95760\") " pod="openshift-ingress-canary/ingress-canary-zc9kr"
Sep 16 17:25:07.011234 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.011077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f7a40a-f3c9-4e73-940e-666151905ce4-config-volume\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:07.011234 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.011104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f7a40a-f3c9-4e73-940e-666151905ce4-tmp-dir\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:07.011234 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.011137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:07.011386 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:07.011247    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:07.011386 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:07.011317    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls podName:a9f7a40a-f3c9-4e73-940e-666151905ce4 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:07.511296645 +0000 UTC m=+36.299919161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls") pod "dns-default-klh2p" (UID: "a9f7a40a-f3c9-4e73-940e-666151905ce4") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:07.011502 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.011399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99nh\" (UniqueName: \"kubernetes.io/projected/a9f7a40a-f3c9-4e73-940e-666151905ce4-kube-api-access-t99nh\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:07.011502 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.011452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swqj5\" (UniqueName: \"kubernetes.io/projected/d9947ef2-8d1a-4b13-b818-b5b2b7d95760-kube-api-access-swqj5\") pod \"ingress-canary-zc9kr\" (UID: \"d9947ef2-8d1a-4b13-b818-b5b2b7d95760\") " pod="openshift-ingress-canary/ingress-canary-zc9kr"
Sep 16 17:25:07.011502 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.011460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f7a40a-f3c9-4e73-940e-666151905ce4-tmp-dir\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:07.011805 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.011783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f7a40a-f3c9-4e73-940e-666151905ce4-config-volume\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:07.013950 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.013928    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9947ef2-8d1a-4b13-b818-b5b2b7d95760-cert\") pod \"ingress-canary-zc9kr\" (UID: \"d9947ef2-8d1a-4b13-b818-b5b2b7d95760\") " pod="openshift-ingress-canary/ingress-canary-zc9kr"
Sep 16 17:25:07.086755 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.086541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t99nh\" (UniqueName: \"kubernetes.io/projected/a9f7a40a-f3c9-4e73-940e-666151905ce4-kube-api-access-t99nh\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:07.087173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.087146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swqj5\" (UniqueName: \"kubernetes.io/projected/d9947ef2-8d1a-4b13-b818-b5b2b7d95760-kube-api-access-swqj5\") pod \"ingress-canary-zc9kr\" (UID: \"d9947ef2-8d1a-4b13-b818-b5b2b7d95760\") " pod="openshift-ingress-canary/ingress-canary-zc9kr"
Sep 16 17:25:07.113523 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.113489    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zc9kr"
Sep 16 17:25:07.516268 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.516173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:07.516444 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:07.516383    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:07.516509 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:07.516452    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls podName:a9f7a40a-f3c9-4e73-940e-666151905ce4 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:08.516430925 +0000 UTC m=+37.305053458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls") pod "dns-default-klh2p" (UID: "a9f7a40a-f3c9-4e73-940e-666151905ce4") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:07.710149 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.710034    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 17:25:07.710984 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.710964    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-trusted-ca\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:07.761578 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.761528    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:25:07.761737 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.761696    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:25:07.770253 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.770193    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 17:25:07.770253 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.770235    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 17:25:07.770450 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.770235    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 17:25:07.770450 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.770368    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qxlnf\""
Sep 16 17:25:07.771125 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.771104    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 17:25:07.783643 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.783609    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-image-registry-private-configuration\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:07.809428 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.809407    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 17:25:07.810440 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:07.810419    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:25:07.810533 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:07.810442    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b89f8fc75-6zrbw: secret "image-registry-tls" not found
Sep 16 17:25:07.810533 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:07.810509    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls podName:d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a nodeName:}" failed. No retries permitted until 2026-09-16 17:25:08.310488249 +0000 UTC m=+37.099110777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls") pod "image-registry-6b89f8fc75-6zrbw" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a") : secret "image-registry-tls" not found
Sep 16 17:25:07.841777 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.841744    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zc9kr"]
Sep 16 17:25:07.846710 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:25:07.846687    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9947ef2_8d1a_4b13_b818_b5b2b7d95760.slice/crio-247108f8c7c5b1394b947a2df9489f96ce8b503e999f0c498838da131138f968 WatchSource:0}: Error finding container 247108f8c7c5b1394b947a2df9489f96ce8b503e999f0c498838da131138f968: Status 404 returned error can't find the container with id 247108f8c7c5b1394b947a2df9489f96ce8b503e999f0c498838da131138f968
Sep 16 17:25:07.910362 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:07.910333    2581 secret.go:189] Couldn't get secret openshift-image-registry/installation-pull-secrets: failed to sync secret cache: timed out waiting for the condition
Sep 16 17:25:07.910473 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:07.910431    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-installation-pull-secrets podName:d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a nodeName:}" failed. No retries permitted until 2026-09-16 17:25:08.410414531 +0000 UTC m=+37.199037042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "installation-pull-secrets" (UniqueName: "kubernetes.io/secret/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-installation-pull-secrets") pod "image-registry-6b89f8fc75-6zrbw" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a") : failed to sync secret cache: timed out waiting for the condition
Sep 16 17:25:07.987224 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.987191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zc9kr" event={"ID":"d9947ef2-8d1a-4b13-b818-b5b2b7d95760","Type":"ContainerStarted","Data":"247108f8c7c5b1394b947a2df9489f96ce8b503e999f0c498838da131138f968"}
Sep 16 17:25:07.990271 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.990246    2581 generic.go:356] "Generic (PLEG): container finished" podID="1daba9ed-cc2c-4e25-b904-1e2a34af7ce1" containerID="77d1952040768c640dc8575e859d7ba5954a12a31f3d429feb33558a3a29dce9" exitCode=0
Sep 16 17:25:07.990391 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:07.990285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nn9tr" event={"ID":"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1","Type":"ContainerDied","Data":"77d1952040768c640dc8575e859d7ba5954a12a31f3d429feb33558a3a29dce9"}
Sep 16 17:25:08.117516 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:08.117489    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 17:25:08.158439 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:08.158403    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gnfmt\""
Sep 16 17:25:08.322079 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:08.321997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:08.322225 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:08.322114    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:25:08.322225 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:08.322127    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b89f8fc75-6zrbw: secret "image-registry-tls" not found
Sep 16 17:25:08.322225 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:08.322181    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls podName:d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a nodeName:}" failed. No retries permitted until 2026-09-16 17:25:09.322164723 +0000 UTC m=+38.110787255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls") pod "image-registry-6b89f8fc75-6zrbw" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a") : secret "image-registry-tls" not found
Sep 16 17:25:08.422701 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:08.422658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-installation-pull-secrets\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:08.425663 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:08.425631    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-installation-pull-secrets\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:08.524002 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:08.523966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:08.524177 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:08.524131    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:08.524239 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:08.524204    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls podName:a9f7a40a-f3c9-4e73-940e-666151905ce4 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:10.524185009 +0000 UTC m=+39.312807520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls") pod "dns-default-klh2p" (UID: "a9f7a40a-f3c9-4e73-940e-666151905ce4") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:08.996713 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:08.996678    2581 generic.go:356] "Generic (PLEG): container finished" podID="1daba9ed-cc2c-4e25-b904-1e2a34af7ce1" containerID="894497c8960c1ac16d8e1cbb0d6c51c3d683807b5738aedc0e79e6a8060977e2" exitCode=0
Sep 16 17:25:08.997217 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:08.996732    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nn9tr" event={"ID":"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1","Type":"ContainerDied","Data":"894497c8960c1ac16d8e1cbb0d6c51c3d683807b5738aedc0e79e6a8060977e2"}
Sep 16 17:25:09.330569 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:09.330318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:09.330754 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:09.330475    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:25:09.330754 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:09.330677    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b89f8fc75-6zrbw: secret "image-registry-tls" not found
Sep 16 17:25:09.330754 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:09.330742    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls podName:d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a nodeName:}" failed. No retries permitted until 2026-09-16 17:25:11.330723136 +0000 UTC m=+40.119345667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls") pod "image-registry-6b89f8fc75-6zrbw" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a") : secret "image-registry-tls" not found
Sep 16 17:25:10.002707 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:10.002613    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nn9tr" event={"ID":"1daba9ed-cc2c-4e25-b904-1e2a34af7ce1","Type":"ContainerStarted","Data":"ae08e1ee1ab6d24392ba78d14bb9c7551320fe21718fa154f8249eab12e2a8d5"}
Sep 16 17:25:10.036826 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:10.036755    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nn9tr" podStartSLOduration=9.070065998 podStartE2EDuration="38.036740213s" totalImagesPullingTime="28.966674215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:10.035468354 +0000 UTC m=+38.824090887" watchObservedRunningTime="2026-09-16 17:25:10.036740213 +0000 UTC m=+38.825362745"
Sep 16 17:25:10.407660 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:10.407626    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:10.520623 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:10.520589    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:10.542593 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:10.542562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:10.542707 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:10.542693    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:10.542761 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:10.542752    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls podName:a9f7a40a-f3c9-4e73-940e-666151905ce4 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:14.542736348 +0000 UTC m=+43.331358863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls") pod "dns-default-klh2p" (UID: "a9f7a40a-f3c9-4e73-940e-666151905ce4") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:11.006343 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:11.006262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zc9kr" event={"ID":"d9947ef2-8d1a-4b13-b818-b5b2b7d95760","Type":"ContainerStarted","Data":"191a35a0b9e848171195dd28d281554595d2733dbe356afef01ad0928af1a56f"}
Sep 16 17:25:11.035301 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:11.035247    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zc9kr" podStartSLOduration=2.677642988 podStartE2EDuration="5.035231088s" totalImagesPullingTime="2.3575881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:11.030106682 +0000 UTC m=+39.818729215" watchObservedRunningTime="2026-09-16 17:25:11.035231088 +0000 UTC m=+39.823853615"
Sep 16 17:25:11.340796 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:11.340761    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:11.348687 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:11.348665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:11.348798 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:11.348786    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:25:11.348850 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:11.348800    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b89f8fc75-6zrbw: secret "image-registry-tls" not found
Sep 16 17:25:11.348881 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:11.348856    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls podName:d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a nodeName:}" failed. No retries permitted until 2026-09-16 17:25:15.348843109 +0000 UTC m=+44.137465620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls") pod "image-registry-6b89f8fc75-6zrbw" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a") : secret "image-registry-tls" not found
Sep 16 17:25:14.190657 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:14.190624    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:14.296731 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:14.296692    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:14.395209 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:14.395170    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:14.464291 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:14.464204    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:14.571264 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:14.571231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:14.571411 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:14.571354    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:14.571411 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:14.571405    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls podName:a9f7a40a-f3c9-4e73-940e-666151905ce4 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:22.571391169 +0000 UTC m=+51.360013681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls") pod "dns-default-klh2p" (UID: "a9f7a40a-f3c9-4e73-940e-666151905ce4") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:15.007185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:15.007153    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:15.089133 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:15.089091    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:15.194741 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:15.194710    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:15.376551 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:15.376514    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:15.376710 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:15.376622    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:25:15.376710 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:15.376635    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b89f8fc75-6zrbw: secret "image-registry-tls" not found
Sep 16 17:25:15.376710 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:15.376686    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls podName:d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a nodeName:}" failed. No retries permitted until 2026-09-16 17:25:23.376672239 +0000 UTC m=+52.165294750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls") pod "image-registry-6b89f8fc75-6zrbw" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a") : secret "image-registry-tls" not found
Sep 16 17:25:16.894345 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:16.894309    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:21.811395 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:21.811360    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:21.895999 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:21.895965    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:22.000620 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:22.000577    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:22.629678 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:22.629633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:22.629858 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:22.629791    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:22.629858 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:22.629855    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls podName:a9f7a40a-f3c9-4e73-940e-666151905ce4 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:38.629838615 +0000 UTC m=+67.418461130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls") pod "dns-default-klh2p" (UID: "a9f7a40a-f3c9-4e73-940e-666151905ce4") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:22.752037 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:22.752002    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:22.852877 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:22.852841    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:22.917444 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:22.917364    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:23.436739 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:23.436700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:23.436926 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:23.436852    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:25:23.436926 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:23.436873    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b89f8fc75-6zrbw: secret "image-registry-tls" not found
Sep 16 17:25:23.437016 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:23.436970    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls podName:d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a nodeName:}" failed. No retries permitted until 2026-09-16 17:25:39.436951823 +0000 UTC m=+68.225574349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls") pod "image-registry-6b89f8fc75-6zrbw" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a") : secret "image-registry-tls" not found
Sep 16 17:25:23.616657 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:23.616626    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:30.025459 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:30.025420    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:30.125197 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:30.125165    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:30.233277 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:30.233243    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:30.471402 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:30.471313    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:30.575663 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:30.575623    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:31.980532 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:31.980504    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kpv5"
Sep 16 17:25:32.149857 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:32.149826    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:32.213523 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:32.213485    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:32.294469 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:32.294435    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:33.556036 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:33.556002    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:33.619472 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:33.619435    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:33.966708 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:33.966621    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:34.027207 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:34.027178    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:35.759590 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:35.759561    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:35.831387 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:35.831354    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:35.901136 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:35.901111    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:37.437803 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:37.437767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs\") pod \"network-metrics-daemon-jhf5t\" (UID: \"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8\") " pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:25:37.442037 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:37.442019    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 17:25:37.448279 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:37.448264    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 17:25:37.448334 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:37.448327    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs podName:fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:41.448305752 +0000 UTC m=+130.236928267 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs") pod "network-metrics-daemon-jhf5t" (UID: "fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8") : secret "metrics-daemon-secret" not found
Sep 16 17:25:37.539023 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:37.538987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:25:37.539101 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:37.539030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hkhb\" (UniqueName: \"kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb\") pod \"network-check-target-ppgtg\" (UID: \"bde6ac75-b65f-4477-829e-4e37b5c55b19\") " pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:25:37.541792 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:37.541776    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 17:25:37.541843 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:37.541777    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 17:25:37.553120 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:37.553093    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 17:25:37.553239 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:37.553204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80100b19-7ac7-4382-af81-3f0a97bd7072-original-pull-secret\") pod \"global-pull-secret-syncer-jxft6\" (UID: \"80100b19-7ac7-4382-af81-3f0a97bd7072\") " pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:25:37.562980 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:37.562953    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hkhb\" (UniqueName: \"kubernetes.io/projected/bde6ac75-b65f-4477-829e-4e37b5c55b19-kube-api-access-2hkhb\") pod \"network-check-target-ppgtg\" (UID: \"bde6ac75-b65f-4477-829e-4e37b5c55b19\") " pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:25:37.771495 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:37.771474    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jxft6"
Sep 16 17:25:37.779658 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:37.779638    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qxlnf\""
Sep 16 17:25:37.787375 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:37.787355    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:25:38.045446 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:38.045369    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jxft6"]
Sep 16 17:25:38.048990 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:25:38.048962    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80100b19_7ac7_4382_af81_3f0a97bd7072.slice/crio-a05222beb91d7c8a7a611b21269f5ff958f355a28374dc3b5c28b3f3ca464fc2 WatchSource:0}: Error finding container a05222beb91d7c8a7a611b21269f5ff958f355a28374dc3b5c28b3f3ca464fc2: Status 404 returned error can't find the container with id a05222beb91d7c8a7a611b21269f5ff958f355a28374dc3b5c28b3f3ca464fc2
Sep 16 17:25:38.065734 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:38.065706    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jxft6" event={"ID":"80100b19-7ac7-4382-af81-3f0a97bd7072","Type":"ContainerStarted","Data":"a05222beb91d7c8a7a611b21269f5ff958f355a28374dc3b5c28b3f3ca464fc2"}
Sep 16 17:25:38.076671 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:38.076646    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ppgtg"]
Sep 16 17:25:38.079591 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:25:38.079570    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde6ac75_b65f_4477_829e_4e37b5c55b19.slice/crio-349c6ade5f2adbdf432036e6e776b3e9677017cd38dd14e347314cc6d83a1044 WatchSource:0}: Error finding container 349c6ade5f2adbdf432036e6e776b3e9677017cd38dd14e347314cc6d83a1044: Status 404 returned error can't find the container with id 349c6ade5f2adbdf432036e6e776b3e9677017cd38dd14e347314cc6d83a1044
Sep 16 17:25:38.646219 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:38.646179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:25:38.646704 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:38.646334    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:38.646704 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:38.646418    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls podName:a9f7a40a-f3c9-4e73-940e-666151905ce4 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:10.646395644 +0000 UTC m=+99.435018157 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls") pod "dns-default-klh2p" (UID: "a9f7a40a-f3c9-4e73-940e-666151905ce4") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:39.070608 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:39.070574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ppgtg" event={"ID":"bde6ac75-b65f-4477-829e-4e37b5c55b19","Type":"ContainerStarted","Data":"349c6ade5f2adbdf432036e6e776b3e9677017cd38dd14e347314cc6d83a1044"}
Sep 16 17:25:39.452220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:39.452137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:25:39.452383 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:39.452292    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:25:39.452383 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:39.452310    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b89f8fc75-6zrbw: secret "image-registry-tls" not found
Sep 16 17:25:39.452383 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:25:39.452376    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls podName:d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a nodeName:}" failed. No retries permitted until 2026-09-16 17:26:11.452357711 +0000 UTC m=+100.240980224 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls") pod "image-registry-6b89f8fc75-6zrbw" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a") : secret "image-registry-tls" not found
Sep 16 17:25:39.548374 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:39.548336    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:39.609536 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:39.609488    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:43.081932 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:43.081879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ppgtg" event={"ID":"bde6ac75-b65f-4477-829e-4e37b5c55b19","Type":"ContainerStarted","Data":"714022573e852c799beec4d61e6d87ef1029c421ff1dad9673265521003dcfc3"}
Sep 16 17:25:43.082454 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:43.081979    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:25:43.083349 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:43.083330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jxft6" event={"ID":"80100b19-7ac7-4382-af81-3f0a97bd7072","Type":"ContainerStarted","Data":"934b6555202ee56a9a39a0d51ad1748127ad1404da7e359cf80b6ec32eb96f97"}
Sep 16 17:25:43.127519 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:43.127468    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jxft6" podStartSLOduration=66.642120874 podStartE2EDuration="1m11.127452508s" totalImagesPullingTime="4.485331634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:43.12697192 +0000 UTC m=+71.915594447" watchObservedRunningTime="2026-09-16 17:25:43.127452508 +0000 UTC m=+71.916075040"
Sep 16 17:25:43.127698 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:43.127677    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ppgtg" podStartSLOduration=66.677314384 podStartE2EDuration="1m11.127671446s" totalImagesPullingTime="4.450357062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:43.107264663 +0000 UTC m=+71.895887196" watchObservedRunningTime="2026-09-16 17:25:43.127671446 +0000 UTC m=+71.916293978"
Sep 16 17:25:50.959138 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:50.959098    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:51.021343 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:51.021309    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:59.695964 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:59.695812    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:59.750912 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:59.750854    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:59.804665 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:25:59.804635    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:26:09.749291 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:26:09.749260    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:26:09.850508 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:26:09.850462    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:26:10.672987 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:26:10.672937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:26:10.673170 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:26:10.673082    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:26:10.673170 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:26:10.673166    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls podName:a9f7a40a-f3c9-4e73-940e-666151905ce4 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:14.673150246 +0000 UTC m=+163.461772757 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls") pod "dns-default-klh2p" (UID: "a9f7a40a-f3c9-4e73-940e-666151905ce4") : secret "dns-default-metrics-tls" not found
Sep 16 17:26:11.480610 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:26:11.480569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:26:11.481048 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:26:11.480725    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:26:11.481048 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:26:11.480744    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b89f8fc75-6zrbw: secret "image-registry-tls" not found
Sep 16 17:26:11.481048 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:26:11.480810    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls podName:d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a nodeName:}" failed. No retries permitted until 2026-09-16 17:27:15.48079501 +0000 UTC m=+164.269417522 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls") pod "image-registry-6b89f8fc75-6zrbw" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a") : secret "image-registry-tls" not found
Sep 16 17:26:14.088552 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:26:14.088521    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ppgtg"
Sep 16 17:26:41.488803 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:26:41.488750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs\") pod \"network-metrics-daemon-jhf5t\" (UID: \"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8\") " pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:26:41.489308 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:26:41.488927    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 17:26:41.489308 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:26:41.488996    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs podName:fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8 nodeName:}" failed. No retries permitted until 2026-09-16 17:28:43.488977394 +0000 UTC m=+252.277599919 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs") pod "network-metrics-daemon-jhf5t" (UID: "fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8") : secret "metrics-daemon-secret" not found
Sep 16 17:27:09.681950 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:09.681886    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw" podUID="d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a"
Sep 16 17:27:09.770445 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:09.770410    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-jhf5t" podUID="fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8"
Sep 16 17:27:09.809006 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:09.808971    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-klh2p" podUID="a9f7a40a-f3c9-4e73-940e-666151905ce4"
Sep 16 17:27:10.291087 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:10.291059    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-klh2p"
Sep 16 17:27:10.291087 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:10.291078    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:27:11.858925 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.858874    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-xg8vh"]
Sep 16 17:27:11.861801 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.861771    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:11.862135 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.862097    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"]
Sep 16 17:27:11.864841 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.864824    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"
Sep 16 17:27:11.870102 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.870086    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 17:27:11.870205 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.870092    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 17:27:11.870792 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.870773    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:27:11.870866 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.870777    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:27:11.872317 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.872295    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hd6rk\""
Sep 16 17:27:11.872653 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.872638    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 17:27:11.872775 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.872755    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 17:27:11.872864 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.872849    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 17:27:11.883194 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.883171    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5mdcm\""
Sep 16 17:27:11.883985 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.883967    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-xg8vh"]
Sep 16 17:27:11.893049 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.893030    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 17:27:11.904093 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:11.904066    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"]
Sep 16 17:27:12.003656 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.003618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwhmg\" (UniqueName: \"kubernetes.io/projected/9e3a6183-641d-4047-9b87-6b79f0870b33-kube-api-access-hwhmg\") pod \"console-operator-696884fb9c-xg8vh\" (UID: \"9e3a6183-641d-4047-9b87-6b79f0870b33\") " pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:12.003656 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.003653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4n5lj\" (UID: \"8bda69d5-56d0-4041-9922-aade015c7466\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"
Sep 16 17:27:12.003862 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.003701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7rxn\" (UniqueName: \"kubernetes.io/projected/8bda69d5-56d0-4041-9922-aade015c7466-kube-api-access-h7rxn\") pod \"cluster-samples-operator-86499c89c4-4n5lj\" (UID: \"8bda69d5-56d0-4041-9922-aade015c7466\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"
Sep 16 17:27:12.003862 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.003722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e3a6183-641d-4047-9b87-6b79f0870b33-serving-cert\") pod \"console-operator-696884fb9c-xg8vh\" (UID: \"9e3a6183-641d-4047-9b87-6b79f0870b33\") " pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:12.003862 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.003736    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e3a6183-641d-4047-9b87-6b79f0870b33-trusted-ca\") pod \"console-operator-696884fb9c-xg8vh\" (UID: \"9e3a6183-641d-4047-9b87-6b79f0870b33\") " pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:12.003862 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.003763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e3a6183-641d-4047-9b87-6b79f0870b33-config\") pod \"console-operator-696884fb9c-xg8vh\" (UID: \"9e3a6183-641d-4047-9b87-6b79f0870b33\") " pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:12.083442 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.083410    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"]
Sep 16 17:27:12.086235 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.086220    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"
Sep 16 17:27:12.090958 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.090913    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 17:27:12.091081 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.091031    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 17:27:12.091144 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.091127    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-zbpbw\""
Sep 16 17:27:12.091572 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.091550    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-b7s95"]
Sep 16 17:27:12.094280 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.094263    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-b7s95"
Sep 16 17:27:12.096270 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.096250    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4"]
Sep 16 17:27:12.096722 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.096709    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-b6drl\""
Sep 16 17:27:12.099085 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.099066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4"
Sep 16 17:27:12.103035 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.103017    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:27:12.103602 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.103580    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 17:27:12.103602 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.103601    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 17:27:12.104084 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.104059    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwhmg\" (UniqueName: \"kubernetes.io/projected/9e3a6183-641d-4047-9b87-6b79f0870b33-kube-api-access-hwhmg\") pod \"console-operator-696884fb9c-xg8vh\" (UID: \"9e3a6183-641d-4047-9b87-6b79f0870b33\") " pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:12.104187 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.104096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4n5lj\" (UID: \"8bda69d5-56d0-4041-9922-aade015c7466\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"
Sep 16 17:27:12.104187 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.104155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7rxn\" (UniqueName: \"kubernetes.io/projected/8bda69d5-56d0-4041-9922-aade015c7466-kube-api-access-h7rxn\") pod \"cluster-samples-operator-86499c89c4-4n5lj\" (UID: \"8bda69d5-56d0-4041-9922-aade015c7466\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"
Sep 16 17:27:12.104306 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.104187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e3a6183-641d-4047-9b87-6b79f0870b33-serving-cert\") pod \"console-operator-696884fb9c-xg8vh\" (UID: \"9e3a6183-641d-4047-9b87-6b79f0870b33\") " pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:12.104306 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.104210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e3a6183-641d-4047-9b87-6b79f0870b33-trusted-ca\") pod \"console-operator-696884fb9c-xg8vh\" (UID: \"9e3a6183-641d-4047-9b87-6b79f0870b33\") " pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:12.104306 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.104235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e3a6183-641d-4047-9b87-6b79f0870b33-config\") pod \"console-operator-696884fb9c-xg8vh\" (UID: \"9e3a6183-641d-4047-9b87-6b79f0870b33\") " pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:12.104306 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:12.104270    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 17:27:12.104306 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.104301    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"]
Sep 16 17:27:12.104549 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:12.104334    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls podName:8bda69d5-56d0-4041-9922-aade015c7466 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:12.604316808 +0000 UTC m=+161.392939319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4n5lj" (UID: "8bda69d5-56d0-4041-9922-aade015c7466") : secret "samples-operator-tls" not found
Sep 16 17:27:12.104549 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.104513    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 17:27:12.105027 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.105001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e3a6183-641d-4047-9b87-6b79f0870b33-config\") pod \"console-operator-696884fb9c-xg8vh\" (UID: \"9e3a6183-641d-4047-9b87-6b79f0870b33\") " pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:12.105810 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.105748    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e3a6183-641d-4047-9b87-6b79f0870b33-trusted-ca\") pod \"console-operator-696884fb9c-xg8vh\" (UID: \"9e3a6183-641d-4047-9b87-6b79f0870b33\") " pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:12.109811 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.108594    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mjmp8\""
Sep 16 17:27:12.110815 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.110322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e3a6183-641d-4047-9b87-6b79f0870b33-serving-cert\") pod \"console-operator-696884fb9c-xg8vh\" (UID: \"9e3a6183-641d-4047-9b87-6b79f0870b33\") " pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:12.121952 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.121927    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-b7s95"]
Sep 16 17:27:12.125471 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.125448    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4"]
Sep 16 17:27:12.130233 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.130213    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwhmg\" (UniqueName: \"kubernetes.io/projected/9e3a6183-641d-4047-9b87-6b79f0870b33-kube-api-access-hwhmg\") pod \"console-operator-696884fb9c-xg8vh\" (UID: \"9e3a6183-641d-4047-9b87-6b79f0870b33\") " pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:12.137156 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.137135    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7rxn\" (UniqueName: \"kubernetes.io/projected/8bda69d5-56d0-4041-9922-aade015c7466-kube-api-access-h7rxn\") pod \"cluster-samples-operator-86499c89c4-4n5lj\" (UID: \"8bda69d5-56d0-4041-9922-aade015c7466\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"
Sep 16 17:27:12.170763 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.170728    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:12.204813 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.204779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9prnd\" (UniqueName: \"kubernetes.io/projected/9395826b-e1df-4879-aa43-86e2af895fe8-kube-api-access-9prnd\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xsmh4\" (UID: \"9395826b-e1df-4879-aa43-86e2af895fe8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4"
Sep 16 17:27:12.205003 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.204843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/68021b3d-3129-4295-a6b4-22454fe05184-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-fdpj6\" (UID: \"68021b3d-3129-4295-a6b4-22454fe05184\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"
Sep 16 17:27:12.205003 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.204924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9395826b-e1df-4879-aa43-86e2af895fe8-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xsmh4\" (UID: \"9395826b-e1df-4879-aa43-86e2af895fe8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4"
Sep 16 17:27:12.205003 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.204942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9395826b-e1df-4879-aa43-86e2af895fe8-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xsmh4\" (UID: \"9395826b-e1df-4879-aa43-86e2af895fe8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4"
Sep 16 17:27:12.205003 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.204958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fdpj6\" (UID: \"68021b3d-3129-4295-a6b4-22454fe05184\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"
Sep 16 17:27:12.205003 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.204975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t75v9\" (UniqueName: \"kubernetes.io/projected/2ef4b019-9c21-40d2-b411-f99d8ed0e41d-kube-api-access-t75v9\") pod \"network-check-source-6f8b768455-b7s95\" (UID: \"2ef4b019-9c21-40d2-b411-f99d8ed0e41d\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-b7s95"
Sep 16 17:27:12.290510 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.290467    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:12.305830 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.305800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9395826b-e1df-4879-aa43-86e2af895fe8-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xsmh4\" (UID: \"9395826b-e1df-4879-aa43-86e2af895fe8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4"
Sep 16 17:27:12.305830 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.305832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9395826b-e1df-4879-aa43-86e2af895fe8-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xsmh4\" (UID: \"9395826b-e1df-4879-aa43-86e2af895fe8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4"
Sep 16 17:27:12.306035 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.305851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fdpj6\" (UID: \"68021b3d-3129-4295-a6b4-22454fe05184\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"
Sep 16 17:27:12.306035 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.305878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t75v9\" (UniqueName: \"kubernetes.io/projected/2ef4b019-9c21-40d2-b411-f99d8ed0e41d-kube-api-access-t75v9\") pod \"network-check-source-6f8b768455-b7s95\" (UID: \"2ef4b019-9c21-40d2-b411-f99d8ed0e41d\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-b7s95"
Sep 16 17:27:12.306035 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:12.305989    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 17:27:12.306147 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:12.306053    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert podName:68021b3d-3129-4295-a6b4-22454fe05184 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:12.8060368 +0000 UTC m=+161.594659316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-fdpj6" (UID: "68021b3d-3129-4295-a6b4-22454fe05184") : secret "networking-console-plugin-cert" not found
Sep 16 17:27:12.306147 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.306092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prnd\" (UniqueName: \"kubernetes.io/projected/9395826b-e1df-4879-aa43-86e2af895fe8-kube-api-access-9prnd\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xsmh4\" (UID: \"9395826b-e1df-4879-aa43-86e2af895fe8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4"
Sep 16 17:27:12.306239 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.306206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/68021b3d-3129-4295-a6b4-22454fe05184-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-fdpj6\" (UID: \"68021b3d-3129-4295-a6b4-22454fe05184\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"
Sep 16 17:27:12.306479 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.306456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9395826b-e1df-4879-aa43-86e2af895fe8-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xsmh4\" (UID: \"9395826b-e1df-4879-aa43-86e2af895fe8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4"
Sep 16 17:27:12.306850 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.306832    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/68021b3d-3129-4295-a6b4-22454fe05184-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-fdpj6\" (UID: \"68021b3d-3129-4295-a6b4-22454fe05184\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"
Sep 16 17:27:12.308309 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.308290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9395826b-e1df-4879-aa43-86e2af895fe8-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xsmh4\" (UID: \"9395826b-e1df-4879-aa43-86e2af895fe8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4"
Sep 16 17:27:12.318686 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.318655    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t75v9\" (UniqueName: \"kubernetes.io/projected/2ef4b019-9c21-40d2-b411-f99d8ed0e41d-kube-api-access-t75v9\") pod \"network-check-source-6f8b768455-b7s95\" (UID: \"2ef4b019-9c21-40d2-b411-f99d8ed0e41d\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-b7s95"
Sep 16 17:27:12.320388 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.320371    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9prnd\" (UniqueName: \"kubernetes.io/projected/9395826b-e1df-4879-aa43-86e2af895fe8-kube-api-access-9prnd\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xsmh4\" (UID: \"9395826b-e1df-4879-aa43-86e2af895fe8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4"
Sep 16 17:27:12.402776 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.402696    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-b7s95"
Sep 16 17:27:12.414982 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.414961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4"
Sep 16 17:27:12.447279 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.446594    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-xg8vh"]
Sep 16 17:27:12.450913 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:27:12.450437    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3a6183_641d_4047_9b87_6b79f0870b33.slice/crio-c04d7bd2a1259adb6523cc4e011006f6e4e376a41a414f33a164e68533cd6c89 WatchSource:0}: Error finding container c04d7bd2a1259adb6523cc4e011006f6e4e376a41a414f33a164e68533cd6c89: Status 404 returned error can't find the container with id c04d7bd2a1259adb6523cc4e011006f6e4e376a41a414f33a164e68533cd6c89
Sep 16 17:27:12.608632 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.608594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4n5lj\" (UID: \"8bda69d5-56d0-4041-9922-aade015c7466\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"
Sep 16 17:27:12.608809 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:12.608763    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 17:27:12.608868 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:12.608842    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls podName:8bda69d5-56d0-4041-9922-aade015c7466 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:13.608821838 +0000 UTC m=+162.397444368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4n5lj" (UID: "8bda69d5-56d0-4041-9922-aade015c7466") : secret "samples-operator-tls" not found
Sep 16 17:27:12.677557 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.677533    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-b7s95"]
Sep 16 17:27:12.679796 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:27:12.679763    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef4b019_9c21_40d2_b411_f99d8ed0e41d.slice/crio-dc92bcf528961e8f46c18968ec371b7c297aaaf506becbd87a6eb97f1e69d625 WatchSource:0}: Error finding container dc92bcf528961e8f46c18968ec371b7c297aaaf506becbd87a6eb97f1e69d625: Status 404 returned error can't find the container with id dc92bcf528961e8f46c18968ec371b7c297aaaf506becbd87a6eb97f1e69d625
Sep 16 17:27:12.697590 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.697566    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4"]
Sep 16 17:27:12.705826 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:27:12.705799    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9395826b_e1df_4879_aa43_86e2af895fe8.slice/crio-c2b64300e820d57ada939e082562e79a9f95caba025ea387fe40ce6348c0e306 WatchSource:0}: Error finding container c2b64300e820d57ada939e082562e79a9f95caba025ea387fe40ce6348c0e306: Status 404 returned error can't find the container with id c2b64300e820d57ada939e082562e79a9f95caba025ea387fe40ce6348c0e306
Sep 16 17:27:12.810526 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:12.810494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fdpj6\" (UID: \"68021b3d-3129-4295-a6b4-22454fe05184\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"
Sep 16 17:27:12.810692 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:12.810638    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 17:27:12.810739 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:12.810706    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert podName:68021b3d-3129-4295-a6b4-22454fe05184 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:13.810687254 +0000 UTC m=+162.599309766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-fdpj6" (UID: "68021b3d-3129-4295-a6b4-22454fe05184") : secret "networking-console-plugin-cert" not found
Sep 16 17:27:13.298904 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:13.298848    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4" event={"ID":"9395826b-e1df-4879-aa43-86e2af895fe8","Type":"ContainerStarted","Data":"c2b64300e820d57ada939e082562e79a9f95caba025ea387fe40ce6348c0e306"}
Sep 16 17:27:13.300628 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:13.300603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-b7s95" event={"ID":"2ef4b019-9c21-40d2-b411-f99d8ed0e41d","Type":"ContainerStarted","Data":"2ce10a0d18fd359526334e6e12091f5c6cf6a9cbfebcd0a117c047a7a87f60cf"}
Sep 16 17:27:13.300763 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:13.300645    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-b7s95" event={"ID":"2ef4b019-9c21-40d2-b411-f99d8ed0e41d","Type":"ContainerStarted","Data":"dc92bcf528961e8f46c18968ec371b7c297aaaf506becbd87a6eb97f1e69d625"}
Sep 16 17:27:13.301763 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:13.301733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh" event={"ID":"9e3a6183-641d-4047-9b87-6b79f0870b33","Type":"ContainerStarted","Data":"c04d7bd2a1259adb6523cc4e011006f6e4e376a41a414f33a164e68533cd6c89"}
Sep 16 17:27:13.352132 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:13.352074    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-b7s95" podStartSLOduration=1.352058397 podStartE2EDuration="1.352058397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:13.351741405 +0000 UTC m=+162.140363940" watchObservedRunningTime="2026-09-16 17:27:13.352058397 +0000 UTC m=+162.140680929"
Sep 16 17:27:13.587562 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:13.587462    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:13.618028 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:13.617997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4n5lj\" (UID: \"8bda69d5-56d0-4041-9922-aade015c7466\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"
Sep 16 17:27:13.618213 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:13.618136    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 17:27:13.618213 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:13.618201    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls podName:8bda69d5-56d0-4041-9922-aade015c7466 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:15.618185381 +0000 UTC m=+164.406807909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4n5lj" (UID: "8bda69d5-56d0-4041-9922-aade015c7466") : secret "samples-operator-tls" not found
Sep 16 17:27:13.819225 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:13.819174    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fdpj6\" (UID: \"68021b3d-3129-4295-a6b4-22454fe05184\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"
Sep 16 17:27:13.819413 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:13.819347    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 17:27:13.819473 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:13.819429    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert podName:68021b3d-3129-4295-a6b4-22454fe05184 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:15.81940831 +0000 UTC m=+164.608030824 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-fdpj6" (UID: "68021b3d-3129-4295-a6b4-22454fe05184") : secret "networking-console-plugin-cert" not found
Sep 16 17:27:14.202711 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:14.202675    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:14.266805 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:14.266770    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:14.726006 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:14.725969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:27:14.726445 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:14.726132    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:27:14.726445 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:14.726196    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls podName:a9f7a40a-f3c9-4e73-940e-666151905ce4 nodeName:}" failed. No retries permitted until 2026-09-16 17:29:16.726176239 +0000 UTC m=+285.514798755 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls") pod "dns-default-klh2p" (UID: "a9f7a40a-f3c9-4e73-940e-666151905ce4") : secret "dns-default-metrics-tls" not found
Sep 16 17:27:15.309383 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:15.309355    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/0.log"
Sep 16 17:27:15.309525 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:15.309410    2581 generic.go:356] "Generic (PLEG): container finished" podID="9e3a6183-641d-4047-9b87-6b79f0870b33" containerID="f6867a8cf9301e6ffaa9fe70f16f4de7a0273acb31a1c66ab2d44197e234dfdc" exitCode=255
Sep 16 17:27:15.309525 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:15.309445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh" event={"ID":"9e3a6183-641d-4047-9b87-6b79f0870b33","Type":"ContainerDied","Data":"f6867a8cf9301e6ffaa9fe70f16f4de7a0273acb31a1c66ab2d44197e234dfdc"}
Sep 16 17:27:15.309721 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:15.309703    2581 scope.go:117] "RemoveContainer" containerID="f6867a8cf9301e6ffaa9fe70f16f4de7a0273acb31a1c66ab2d44197e234dfdc"
Sep 16 17:27:15.531709 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:15.531613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls\") pod \"image-registry-6b89f8fc75-6zrbw\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") " pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:27:15.531864 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:15.531769    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:27:15.531864 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:15.531788    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b89f8fc75-6zrbw: secret "image-registry-tls" not found
Sep 16 17:27:15.531864 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:15.531860    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls podName:d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a nodeName:}" failed. No retries permitted until 2026-09-16 17:29:17.531844012 +0000 UTC m=+286.320466524 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls") pod "image-registry-6b89f8fc75-6zrbw" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a") : secret "image-registry-tls" not found
Sep 16 17:27:15.632996 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:15.632967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4n5lj\" (UID: \"8bda69d5-56d0-4041-9922-aade015c7466\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"
Sep 16 17:27:15.633134 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:15.633114    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 17:27:15.633193 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:15.633184    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls podName:8bda69d5-56d0-4041-9922-aade015c7466 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:19.633169247 +0000 UTC m=+168.421791759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4n5lj" (UID: "8bda69d5-56d0-4041-9922-aade015c7466") : secret "samples-operator-tls" not found
Sep 16 17:27:15.835144 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:15.835044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fdpj6\" (UID: \"68021b3d-3129-4295-a6b4-22454fe05184\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"
Sep 16 17:27:15.835492 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:15.835188    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 17:27:15.835492 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:15.835269    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert podName:68021b3d-3129-4295-a6b4-22454fe05184 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:19.835253955 +0000 UTC m=+168.623876466 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-fdpj6" (UID: "68021b3d-3129-4295-a6b4-22454fe05184") : secret "networking-console-plugin-cert" not found
Sep 16 17:27:16.314598 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.314567    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/1.log"
Sep 16 17:27:16.314992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.314980    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/0.log"
Sep 16 17:27:16.315051 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.315012    2581 generic.go:356] "Generic (PLEG): container finished" podID="9e3a6183-641d-4047-9b87-6b79f0870b33" containerID="3f780cabdf6299ad68a9e4021aa6c61eaa452fbc6288454dbde87e27ed8f2f2b" exitCode=255
Sep 16 17:27:16.315108 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.315052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh" event={"ID":"9e3a6183-641d-4047-9b87-6b79f0870b33","Type":"ContainerDied","Data":"3f780cabdf6299ad68a9e4021aa6c61eaa452fbc6288454dbde87e27ed8f2f2b"}
Sep 16 17:27:16.315190 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.315174    2581 scope.go:117] "RemoveContainer" containerID="f6867a8cf9301e6ffaa9fe70f16f4de7a0273acb31a1c66ab2d44197e234dfdc"
Sep 16 17:27:16.315410 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.315388    2581 scope.go:117] "RemoveContainer" containerID="3f780cabdf6299ad68a9e4021aa6c61eaa452fbc6288454dbde87e27ed8f2f2b"
Sep 16 17:27:16.315650 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:16.315626    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-xg8vh_openshift-console-operator(9e3a6183-641d-4047-9b87-6b79f0870b33)\"" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh" podUID="9e3a6183-641d-4047-9b87-6b79f0870b33"
Sep 16 17:27:16.316606 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.316574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4" event={"ID":"9395826b-e1df-4879-aa43-86e2af895fe8","Type":"ContainerStarted","Data":"e32c5ae44bdb25b482c9d7ad16bc217891e7986863599b4bea5640d787512e98"}
Sep 16 17:27:16.356228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.356175    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4" podStartSLOduration=1.7919752249999998 podStartE2EDuration="4.356155969s" totalImagesPullingTime="2.564180744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:16.355856586 +0000 UTC m=+165.144479102" watchObservedRunningTime="2026-09-16 17:27:16.356155969 +0000 UTC m=+165.144778502"
Sep 16 17:27:16.405230 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.405200    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:16.723571 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.723495    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n88th"]
Sep 16 17:27:16.724792 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.724757    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:16.727752 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.727734    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:16.731906 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.731866    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 17:27:16.732017 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.731870    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 17:27:16.733084 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.733068    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 17:27:16.733084 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.733079    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 17:27:16.734332 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.734317    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r5z4f\""
Sep 16 17:27:16.746971 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.746947    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n88th"]
Sep 16 17:27:16.794931 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.794862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:16.843166 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.843127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkrg2\" (UniqueName: \"kubernetes.io/projected/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-api-access-qkrg2\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:16.843541 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.843226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d87112b0-92d4-41bc-985f-572459ec8c6e-data-volume\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:16.843541 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.843264    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:16.843541 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.843303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:16.843541 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.843398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d87112b0-92d4-41bc-985f-572459ec8c6e-crio-socket\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:16.944352 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.944307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:16.944352 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.944357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:16.944594 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.944412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d87112b0-92d4-41bc-985f-572459ec8c6e-crio-socket\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:16.944594 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.944446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkrg2\" (UniqueName: \"kubernetes.io/projected/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-api-access-qkrg2\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:16.944594 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:16.944453    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:16.944594 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:16.944507    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:16.944594 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.944516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d87112b0-92d4-41bc-985f-572459ec8c6e-data-volume\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:16.944594 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.944524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d87112b0-92d4-41bc-985f-572459ec8c6e-crio-socket\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:16.944594 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:16.944541    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls podName:d87112b0-92d4-41bc-985f-572459ec8c6e nodeName:}" failed. No retries permitted until 2026-09-16 17:27:17.444520847 +0000 UTC m=+166.233143360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n88th" (UID: "d87112b0-92d4-41bc-985f-572459ec8c6e") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:16.944594 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:16.944570    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm podName:d87112b0-92d4-41bc-985f-572459ec8c6e nodeName:}" failed. No retries permitted until 2026-09-16 17:27:17.444552764 +0000 UTC m=+166.233175295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n88th" (UID: "d87112b0-92d4-41bc-985f-572459ec8c6e") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:16.944983 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.944824    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d87112b0-92d4-41bc-985f-572459ec8c6e-data-volume\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:16.960872 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:16.960835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkrg2\" (UniqueName: \"kubernetes.io/projected/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-api-access-qkrg2\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:17.321716 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.321688    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/1.log"
Sep 16 17:27:17.322071 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.322053    2581 scope.go:117] "RemoveContainer" containerID="3f780cabdf6299ad68a9e4021aa6c61eaa452fbc6288454dbde87e27ed8f2f2b"
Sep 16 17:27:17.322261 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:17.322240    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-xg8vh_openshift-console-operator(9e3a6183-641d-4047-9b87-6b79f0870b33)\"" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh" podUID="9e3a6183-641d-4047-9b87-6b79f0870b33"
Sep 16 17:27:17.333401 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.333365    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:17.378607 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.378577    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-swmsn"]
Sep 16 17:27:17.382112 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.382095    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-swmsn"
Sep 16 17:27:17.388340 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.388276    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2sj97\""
Sep 16 17:27:17.388597 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.388570    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 17:27:17.388656 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.388608    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 17:27:17.398681 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.398656    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-swmsn"]
Sep 16 17:27:17.420048 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.419996    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:17.448156 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.448125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:17.448291 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:17.448251    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:17.448371 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:17.448329    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm podName:d87112b0-92d4-41bc-985f-572459ec8c6e nodeName:}" failed. No retries permitted until 2026-09-16 17:27:18.448307371 +0000 UTC m=+167.236929886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n88th" (UID: "d87112b0-92d4-41bc-985f-572459ec8c6e") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:17.448702 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.448399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:17.448702 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:17.448502    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:17.448702 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:17.448540    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls podName:d87112b0-92d4-41bc-985f-572459ec8c6e nodeName:}" failed. No retries permitted until 2026-09-16 17:27:18.448528775 +0000 UTC m=+167.237151286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n88th" (UID: "d87112b0-92d4-41bc-985f-572459ec8c6e") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:17.525045 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.525010    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:17.549378 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.549341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dblnf\" (UniqueName: \"kubernetes.io/projected/df97691c-0462-40ce-b67c-944b2e45ef2c-kube-api-access-dblnf\") pod \"migrator-8b4d697c5-swmsn\" (UID: \"df97691c-0462-40ce-b67c-944b2e45ef2c\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-swmsn"
Sep 16 17:27:17.650256 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.650152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dblnf\" (UniqueName: \"kubernetes.io/projected/df97691c-0462-40ce-b67c-944b2e45ef2c-kube-api-access-dblnf\") pod \"migrator-8b4d697c5-swmsn\" (UID: \"df97691c-0462-40ce-b67c-944b2e45ef2c\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-swmsn"
Sep 16 17:27:17.665048 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.665019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dblnf\" (UniqueName: \"kubernetes.io/projected/df97691c-0462-40ce-b67c-944b2e45ef2c-kube-api-access-dblnf\") pod \"migrator-8b4d697c5-swmsn\" (UID: \"df97691c-0462-40ce-b67c-944b2e45ef2c\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-swmsn"
Sep 16 17:27:17.690820 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.690789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-swmsn"
Sep 16 17:27:17.973197 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:17.973171    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-swmsn"]
Sep 16 17:27:17.975224 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:27:17.975195    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf97691c_0462_40ce_b67c_944b2e45ef2c.slice/crio-35adbeb81588e8f5eae459498c8e4979baf76ca24b0e0a88306d154381d7df64 WatchSource:0}: Error finding container 35adbeb81588e8f5eae459498c8e4979baf76ca24b0e0a88306d154381d7df64: Status 404 returned error can't find the container with id 35adbeb81588e8f5eae459498c8e4979baf76ca24b0e0a88306d154381d7df64
Sep 16 17:27:18.330620 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:18.330569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-swmsn" event={"ID":"df97691c-0462-40ce-b67c-944b2e45ef2c","Type":"ContainerStarted","Data":"35adbeb81588e8f5eae459498c8e4979baf76ca24b0e0a88306d154381d7df64"}
Sep 16 17:27:18.457233 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:18.457197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:18.457233 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:18.457240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:18.457482 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:18.457333    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:18.457482 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:18.457369    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:18.457482 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:18.457398    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm podName:d87112b0-92d4-41bc-985f-572459ec8c6e nodeName:}" failed. No retries permitted until 2026-09-16 17:27:20.45738112 +0000 UTC m=+169.246003634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n88th" (UID: "d87112b0-92d4-41bc-985f-572459ec8c6e") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:18.457482 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:18.457436    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls podName:d87112b0-92d4-41bc-985f-572459ec8c6e nodeName:}" failed. No retries permitted until 2026-09-16 17:27:20.457417134 +0000 UTC m=+169.246039659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n88th" (UID: "d87112b0-92d4-41bc-985f-572459ec8c6e") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:18.485240 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:18.485206    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7rg26_22254a42-55b7-4718-abf3-475178a65fc9/dns-node-resolver/0.log"
Sep 16 17:27:18.774420 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:18.774374    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:18.832230 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:18.832190    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:19.205844 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:19.205752    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:19.279123 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:19.279083    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:19.335392 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:19.335359    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-swmsn" event={"ID":"df97691c-0462-40ce-b67c-944b2e45ef2c","Type":"ContainerStarted","Data":"1a7a6926819862174574b0661094d4427632f1566d3d0f88c5995ea08fac4468"}
Sep 16 17:27:19.335525 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:19.335409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-swmsn" event={"ID":"df97691c-0462-40ce-b67c-944b2e45ef2c","Type":"ContainerStarted","Data":"041b7f8916d2204eeb2f8807689ca0a784de07b05ffdd5bc3ceb8c88d2154a50"}
Sep 16 17:27:19.362514 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:19.362468    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-swmsn" podStartSLOduration=1.504400344 podStartE2EDuration="2.362440643s" totalImagesPullingTime="858.040299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:19.360917359 +0000 UTC m=+168.149539886" watchObservedRunningTime="2026-09-16 17:27:19.362440643 +0000 UTC m=+168.151063175"
Sep 16 17:27:19.473205 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:19.473125    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:19.573324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:19.573296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:19.646474 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:19.646454    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qzm9m_5638697b-24e8-4230-857d-7add1666f79a/node-ca/0.log"
Sep 16 17:27:19.667846 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:19.667804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4n5lj\" (UID: \"8bda69d5-56d0-4041-9922-aade015c7466\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"
Sep 16 17:27:19.668005 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:19.667964    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 17:27:19.668048 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:19.668027    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls podName:8bda69d5-56d0-4041-9922-aade015c7466 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:27.668011376 +0000 UTC m=+176.456633887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4n5lj" (UID: "8bda69d5-56d0-4041-9922-aade015c7466") : secret "samples-operator-tls" not found
Sep 16 17:27:19.870132 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:19.870098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fdpj6\" (UID: \"68021b3d-3129-4295-a6b4-22454fe05184\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"
Sep 16 17:27:19.870297 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:19.870242    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 17:27:19.870338 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:19.870306    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert podName:68021b3d-3129-4295-a6b4-22454fe05184 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:27.870290039 +0000 UTC m=+176.658912554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-fdpj6" (UID: "68021b3d-3129-4295-a6b4-22454fe05184") : secret "networking-console-plugin-cert" not found
Sep 16 17:27:20.475912 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:20.475842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:20.475912 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:20.475917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:20.476381 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:20.475983    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:20.476381 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:20.476010    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:20.476381 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:20.476033    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm podName:d87112b0-92d4-41bc-985f-572459ec8c6e nodeName:}" failed. No retries permitted until 2026-09-16 17:27:24.476020505 +0000 UTC m=+173.264643017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n88th" (UID: "d87112b0-92d4-41bc-985f-572459ec8c6e") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:20.476381 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:20.476059    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls podName:d87112b0-92d4-41bc-985f-572459ec8c6e nodeName:}" failed. No retries permitted until 2026-09-16 17:27:24.476047551 +0000 UTC m=+173.264670063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n88th" (UID: "d87112b0-92d4-41bc-985f-572459ec8c6e") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:20.642505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:20.642470    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zc9kr_d9947ef2-8d1a-4b13-b818-b5b2b7d95760/serve-healthcheck-canary/0.log"
Sep 16 17:27:21.081867 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.081832    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rlzjh"]
Sep 16 17:27:21.084835 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.084820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-rlzjh"
Sep 16 17:27:21.087659 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.087635    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9j57n\""
Sep 16 17:27:21.087778 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.087671    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 17:27:21.087778 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.087677    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 17:27:21.088698 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.088682    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 17:27:21.088842 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.088826    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 17:27:21.100267 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.100244    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rlzjh"]
Sep 16 17:27:21.136911 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.136862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:21.180324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.180286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9303cde9-abf7-4031-ad25-dcfb56787aaf-signing-key\") pod \"service-ca-568869fdc5-rlzjh\" (UID: \"9303cde9-abf7-4031-ad25-dcfb56787aaf\") " pod="openshift-service-ca/service-ca-568869fdc5-rlzjh"
Sep 16 17:27:21.180460 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.180354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6dv5\" (UniqueName: \"kubernetes.io/projected/9303cde9-abf7-4031-ad25-dcfb56787aaf-kube-api-access-z6dv5\") pod \"service-ca-568869fdc5-rlzjh\" (UID: \"9303cde9-abf7-4031-ad25-dcfb56787aaf\") " pod="openshift-service-ca/service-ca-568869fdc5-rlzjh"
Sep 16 17:27:21.180460 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.180403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9303cde9-abf7-4031-ad25-dcfb56787aaf-signing-cabundle\") pod \"service-ca-568869fdc5-rlzjh\" (UID: \"9303cde9-abf7-4031-ad25-dcfb56787aaf\") " pod="openshift-service-ca/service-ca-568869fdc5-rlzjh"
Sep 16 17:27:21.237270 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.237234    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:21.281524 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.281481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6dv5\" (UniqueName: \"kubernetes.io/projected/9303cde9-abf7-4031-ad25-dcfb56787aaf-kube-api-access-z6dv5\") pod \"service-ca-568869fdc5-rlzjh\" (UID: \"9303cde9-abf7-4031-ad25-dcfb56787aaf\") " pod="openshift-service-ca/service-ca-568869fdc5-rlzjh"
Sep 16 17:27:21.281658 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.281560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9303cde9-abf7-4031-ad25-dcfb56787aaf-signing-cabundle\") pod \"service-ca-568869fdc5-rlzjh\" (UID: \"9303cde9-abf7-4031-ad25-dcfb56787aaf\") " pod="openshift-service-ca/service-ca-568869fdc5-rlzjh"
Sep 16 17:27:21.281711 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.281656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9303cde9-abf7-4031-ad25-dcfb56787aaf-signing-key\") pod \"service-ca-568869fdc5-rlzjh\" (UID: \"9303cde9-abf7-4031-ad25-dcfb56787aaf\") " pod="openshift-service-ca/service-ca-568869fdc5-rlzjh"
Sep 16 17:27:21.282228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.282206    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9303cde9-abf7-4031-ad25-dcfb56787aaf-signing-cabundle\") pod \"service-ca-568869fdc5-rlzjh\" (UID: \"9303cde9-abf7-4031-ad25-dcfb56787aaf\") " pod="openshift-service-ca/service-ca-568869fdc5-rlzjh"
Sep 16 17:27:21.284249 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.284224    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9303cde9-abf7-4031-ad25-dcfb56787aaf-signing-key\") pod \"service-ca-568869fdc5-rlzjh\" (UID: \"9303cde9-abf7-4031-ad25-dcfb56787aaf\") " pod="openshift-service-ca/service-ca-568869fdc5-rlzjh"
Sep 16 17:27:21.291207 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.291183    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6dv5\" (UniqueName: \"kubernetes.io/projected/9303cde9-abf7-4031-ad25-dcfb56787aaf-kube-api-access-z6dv5\") pod \"service-ca-568869fdc5-rlzjh\" (UID: \"9303cde9-abf7-4031-ad25-dcfb56787aaf\") " pod="openshift-service-ca/service-ca-568869fdc5-rlzjh"
Sep 16 17:27:21.393383 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.393289    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-rlzjh"
Sep 16 17:27:21.666632 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:21.666535    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rlzjh"]
Sep 16 17:27:21.669496 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:27:21.669468    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9303cde9_abf7_4031_ad25_dcfb56787aaf.slice/crio-15a45708e2c5c918a4b82363d0fefd097ce904ec7a856bf2b359d05c0a54a8b7 WatchSource:0}: Error finding container 15a45708e2c5c918a4b82363d0fefd097ce904ec7a856bf2b359d05c0a54a8b7: Status 404 returned error can't find the container with id 15a45708e2c5c918a4b82363d0fefd097ce904ec7a856bf2b359d05c0a54a8b7
Sep 16 17:27:22.150071 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:22.150027    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:22.171829 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:22.171800    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:22.172012 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:22.171912    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:22.172378 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:22.172359    2581 scope.go:117] "RemoveContainer" containerID="3f780cabdf6299ad68a9e4021aa6c61eaa452fbc6288454dbde87e27ed8f2f2b"
Sep 16 17:27:22.172580 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:22.172552    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-xg8vh_openshift-console-operator(9e3a6183-641d-4047-9b87-6b79f0870b33)\"" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh" podUID="9e3a6183-641d-4047-9b87-6b79f0870b33"
Sep 16 17:27:22.346498 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:22.346456    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-rlzjh" event={"ID":"9303cde9-abf7-4031-ad25-dcfb56787aaf","Type":"ContainerStarted","Data":"15a45708e2c5c918a4b82363d0fefd097ce904ec7a856bf2b359d05c0a54a8b7"}
Sep 16 17:27:24.354378 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:24.354348    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-rlzjh" event={"ID":"9303cde9-abf7-4031-ad25-dcfb56787aaf","Type":"ContainerStarted","Data":"de6b7ae64692052cb1a1b4609c15ff5950341556b298319a950300f2b2fff18a"}
Sep 16 17:27:24.373338 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:24.373286    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-rlzjh" podStartSLOduration=1.447559863 podStartE2EDuration="3.373269773s" totalImagesPullingTime="1.92570991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:24.372817747 +0000 UTC m=+173.161440282" watchObservedRunningTime="2026-09-16 17:27:24.373269773 +0000 UTC m=+173.161892291"
Sep 16 17:27:24.507503 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:24.507463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:24.507667 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:24.507510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:24.507667 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:24.507610    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:24.507667 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:24.507640    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:24.507769 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:24.507672    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm podName:d87112b0-92d4-41bc-985f-572459ec8c6e nodeName:}" failed. No retries permitted until 2026-09-16 17:27:32.507655632 +0000 UTC m=+181.296278173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n88th" (UID: "d87112b0-92d4-41bc-985f-572459ec8c6e") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:24.507769 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:24.507704    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls podName:d87112b0-92d4-41bc-985f-572459ec8c6e nodeName:}" failed. No retries permitted until 2026-09-16 17:27:32.507686296 +0000 UTC m=+181.296308812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n88th" (UID: "d87112b0-92d4-41bc-985f-572459ec8c6e") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:24.760938 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:24.760853    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:27:24.933626 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:24.933598    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:25.036822 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:25.036789    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:25.133658 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:25.133625    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:25.234594 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:25.234561    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:25.344112 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:25.344015    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:25.828854 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:25.828821    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:25.991861 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:25.991816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:26.642066 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:26.642030    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:26.837864 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:26.837831    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:27.037491 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:27.037456    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:27.240460 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:27.240420    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:27.734070 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:27.734028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4n5lj\" (UID: \"8bda69d5-56d0-4041-9922-aade015c7466\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"
Sep 16 17:27:27.736661 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:27.736635    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bda69d5-56d0-4041-9922-aade015c7466-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4n5lj\" (UID: \"8bda69d5-56d0-4041-9922-aade015c7466\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"
Sep 16 17:27:27.775565 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:27.775542    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"
Sep 16 17:27:27.935970 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:27.935928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fdpj6\" (UID: \"68021b3d-3129-4295-a6b4-22454fe05184\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"
Sep 16 17:27:27.936353 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:27.936082    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 17:27:27.936353 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:27.936153    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert podName:68021b3d-3129-4295-a6b4-22454fe05184 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:43.936137072 +0000 UTC m=+192.724759583 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-fdpj6" (UID: "68021b3d-3129-4295-a6b4-22454fe05184") : secret "networking-console-plugin-cert" not found
Sep 16 17:27:28.060114 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:28.060091    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj"]
Sep 16 17:27:28.079006 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:28.078944    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:28.245401 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:28.245366    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:28.372865 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:28.372774    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj" event={"ID":"8bda69d5-56d0-4041-9922-aade015c7466","Type":"ContainerStarted","Data":"82c43ad6a937a8488ba1f5cd08da7da29f420a012642b535e969d79a9f526155"}
Sep 16 17:27:28.811017 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:28.810961    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:28.916355 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:28.916305    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:29.242546 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:29.242452    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:29.697064 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:29.697032    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:29.816722 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:29.816683    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:29.924630 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:29.924545    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:29.996557 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:29.996519    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:30.260364 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:30.260271    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:30.381425 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:30.381394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj" event={"ID":"8bda69d5-56d0-4041-9922-aade015c7466","Type":"ContainerStarted","Data":"5116c35d73420268f6355a92e01cb73871bbe2efd25703355f7712534860e56f"}
Sep 16 17:27:30.381425 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:30.381443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj" event={"ID":"8bda69d5-56d0-4041-9922-aade015c7466","Type":"ContainerStarted","Data":"01a6a64693043f6f9a806c6c42a993f6db42c252cf0c287a51d2062d614758f6"}
Sep 16 17:27:30.411325 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:30.411274    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4n5lj" podStartSLOduration=17.926791067 podStartE2EDuration="19.411258113s" totalImagesPullingTime="1.484467046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:30.410859064 +0000 UTC m=+179.199481595" watchObservedRunningTime="2026-09-16 17:27:30.411258113 +0000 UTC m=+179.199880709"
Sep 16 17:27:31.028437 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:31.028398    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:31.402121 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:31.402037    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:31.499519 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:31.499474    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:31.640962 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:31.640926    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:31.831763 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:31.831724    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:32.225884 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:32.225799    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:32.581052 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:32.581019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:32.581515 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:32.581153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:32.581515 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:32.581275    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:32.581515 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:32.581349    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls podName:d87112b0-92d4-41bc-985f-572459ec8c6e nodeName:}" failed. No retries permitted until 2026-09-16 17:27:48.581327007 +0000 UTC m=+197.369949540 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n88th" (UID: "d87112b0-92d4-41bc-985f-572459ec8c6e") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:32.581659 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:32.581518    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d87112b0-92d4-41bc-985f-572459ec8c6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:32.845962 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:32.845864    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:33.429343 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:33.429311    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:33.632366 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:33.632325    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:33.837364 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:33.837276    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:34.031930 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:34.031836    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:34.228434 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:34.228329    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:34.761171 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:34.761137    2581 scope.go:117] "RemoveContainer" containerID="3f780cabdf6299ad68a9e4021aa6c61eaa452fbc6288454dbde87e27ed8f2f2b"
Sep 16 17:27:34.837424 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:34.837395    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:35.233946 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:35.233914    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:35.399596 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:35.399567    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:27:35.400009 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:35.399995    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/1.log"
Sep 16 17:27:35.400088 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:35.400026    2581 generic.go:356] "Generic (PLEG): container finished" podID="9e3a6183-641d-4047-9b87-6b79f0870b33" containerID="aa4316cd5f7e45b37627defd614f493ec4944527758776c9c198fff1c47be2a2" exitCode=255
Sep 16 17:27:35.400088 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:35.400069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh" event={"ID":"9e3a6183-641d-4047-9b87-6b79f0870b33","Type":"ContainerDied","Data":"aa4316cd5f7e45b37627defd614f493ec4944527758776c9c198fff1c47be2a2"}
Sep 16 17:27:35.400150 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:35.400113    2581 scope.go:117] "RemoveContainer" containerID="3f780cabdf6299ad68a9e4021aa6c61eaa452fbc6288454dbde87e27ed8f2f2b"
Sep 16 17:27:35.400426 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:35.400410    2581 scope.go:117] "RemoveContainer" containerID="aa4316cd5f7e45b37627defd614f493ec4944527758776c9c198fff1c47be2a2"
Sep 16 17:27:35.400584 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:35.400567    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-xg8vh_openshift-console-operator(9e3a6183-641d-4047-9b87-6b79f0870b33)\"" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh" podUID="9e3a6183-641d-4047-9b87-6b79f0870b33"
Sep 16 17:27:35.449390 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:35.449352    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:35.631835 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:35.631795    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:36.404391 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:36.404367    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:27:42.171772 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:42.171732    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:42.172185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:42.171785    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:27:42.172185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:42.172101    2581 scope.go:117] "RemoveContainer" containerID="aa4316cd5f7e45b37627defd614f493ec4944527758776c9c198fff1c47be2a2"
Sep 16 17:27:42.172270 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:42.172240    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-xg8vh_openshift-console-operator(9e3a6183-641d-4047-9b87-6b79f0870b33)\"" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh" podUID="9e3a6183-641d-4047-9b87-6b79f0870b33"
Sep 16 17:27:43.968845 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:43.968787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fdpj6\" (UID: \"68021b3d-3129-4295-a6b4-22454fe05184\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"
Sep 16 17:27:43.971281 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:43.971260    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/68021b3d-3129-4295-a6b4-22454fe05184-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fdpj6\" (UID: \"68021b3d-3129-4295-a6b4-22454fe05184\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"
Sep 16 17:27:44.197928 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:44.197876    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-zbpbw\""
Sep 16 17:27:44.205288 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:44.205270    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"
Sep 16 17:27:44.484436 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:44.484370    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6"]
Sep 16 17:27:44.486817 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:27:44.486786    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68021b3d_3129_4295_a6b4_22454fe05184.slice/crio-89ea77433af28e24df7ee02e9c39229d80e43debe066b74a87a7ee96e91485f7 WatchSource:0}: Error finding container 89ea77433af28e24df7ee02e9c39229d80e43debe066b74a87a7ee96e91485f7: Status 404 returned error can't find the container with id 89ea77433af28e24df7ee02e9c39229d80e43debe066b74a87a7ee96e91485f7
Sep 16 17:27:44.487495 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:44.487445    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:44.595429 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:44.595398    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:44.786716 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:44.786681    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:44.865019 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:44.864933    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:45.433628 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:45.433585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6" event={"ID":"68021b3d-3129-4295-a6b4-22454fe05184","Type":"ContainerStarted","Data":"89ea77433af28e24df7ee02e9c39229d80e43debe066b74a87a7ee96e91485f7"}
Sep 16 17:27:46.343810 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.343779    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:46.438590 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.438559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6" event={"ID":"68021b3d-3129-4295-a6b4-22454fe05184","Type":"ContainerStarted","Data":"35c2b25159b6f08c891d3ed1506a0fb37a73c6817a5935b2b76947713c9af61c"}
Sep 16 17:27:46.441144 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.441108    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:46.463225 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.463169    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fdpj6" podStartSLOduration=33.265931688 podStartE2EDuration="34.463153342s" totalImagesPullingTime="1.197221654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:46.461628713 +0000 UTC m=+195.250251246" watchObservedRunningTime="2026-09-16 17:27:46.463153342 +0000 UTC m=+195.251775875"
Sep 16 17:27:46.546762 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.546730    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:46.577060 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.577026    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fcxsr"]
Sep 16 17:27:46.579984 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.579969    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fcxsr"
Sep 16 17:27:46.584123 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.584103    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 17:27:46.584208 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.584162    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kd4hb\""
Sep 16 17:27:46.594142 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.594083    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fcxsr"]
Sep 16 17:27:46.626178 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.626143    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:46.691340 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.691302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/81d5aa9b-3cae-4ac7-a5fc-d2c1729f14fb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-fcxsr\" (UID: \"81d5aa9b-3cae-4ac7-a5fc-d2c1729f14fb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fcxsr"
Sep 16 17:27:46.707548 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.707507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:46.791990 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.791951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/81d5aa9b-3cae-4ac7-a5fc-d2c1729f14fb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-fcxsr\" (UID: \"81d5aa9b-3cae-4ac7-a5fc-d2c1729f14fb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fcxsr"
Sep 16 17:27:46.792160 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:46.792107    2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 17:27:46.792200 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:46.792170    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81d5aa9b-3cae-4ac7-a5fc-d2c1729f14fb-tls-certificates podName:81d5aa9b-3cae-4ac7-a5fc-d2c1729f14fb nodeName:}" failed. No retries permitted until 2026-09-16 17:27:47.292154412 +0000 UTC m=+196.080776923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/81d5aa9b-3cae-4ac7-a5fc-d2c1729f14fb-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-fcxsr" (UID: "81d5aa9b-3cae-4ac7-a5fc-d2c1729f14fb") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 17:27:46.803996 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.803964    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:46.897542 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:46.897453    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:47.295761 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:47.295728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/81d5aa9b-3cae-4ac7-a5fc-d2c1729f14fb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-fcxsr\" (UID: \"81d5aa9b-3cae-4ac7-a5fc-d2c1729f14fb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fcxsr"
Sep 16 17:27:47.298200 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:47.298182    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/81d5aa9b-3cae-4ac7-a5fc-d2c1729f14fb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-fcxsr\" (UID: \"81d5aa9b-3cae-4ac7-a5fc-d2c1729f14fb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fcxsr"
Sep 16 17:27:47.488123 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:47.488088    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fcxsr"
Sep 16 17:27:47.758227 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:47.758199    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fcxsr"]
Sep 16 17:27:47.760393 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:27:47.760368    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d5aa9b_3cae_4ac7_a5fc_d2c1729f14fb.slice/crio-24fe2c96c02d3653321069e62331c2275e6b340d396d7694aaf54b5e550d01b7 WatchSource:0}: Error finding container 24fe2c96c02d3653321069e62331c2275e6b340d396d7694aaf54b5e550d01b7: Status 404 returned error can't find the container with id 24fe2c96c02d3653321069e62331c2275e6b340d396d7694aaf54b5e550d01b7
Sep 16 17:27:48.446621 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:48.446575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fcxsr" event={"ID":"81d5aa9b-3cae-4ac7-a5fc-d2c1729f14fb","Type":"ContainerStarted","Data":"24fe2c96c02d3653321069e62331c2275e6b340d396d7694aaf54b5e550d01b7"}
Sep 16 17:27:48.605814 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:48.605776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:48.608441 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:48.608413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d87112b0-92d4-41bc-985f-572459ec8c6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n88th\" (UID: \"d87112b0-92d4-41bc-985f-572459ec8c6e\") " pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:48.840968 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:48.840936    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r5z4f\""
Sep 16 17:27:48.846439 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:48.846417    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n88th"
Sep 16 17:27:49.139464 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.139439    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n88th"]
Sep 16 17:27:49.140482 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:27:49.140457    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87112b0_92d4_41bc_985f_572459ec8c6e.slice/crio-e5b0e15bdf277b7a1404e8fd1a49ad37d75b1f95a0ad74068b123a07c48764d5 WatchSource:0}: Error finding container e5b0e15bdf277b7a1404e8fd1a49ad37d75b1f95a0ad74068b123a07c48764d5: Status 404 returned error can't find the container with id e5b0e15bdf277b7a1404e8fd1a49ad37d75b1f95a0ad74068b123a07c48764d5
Sep 16 17:27:49.451386 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.451289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fcxsr" event={"ID":"81d5aa9b-3cae-4ac7-a5fc-d2c1729f14fb","Type":"ContainerStarted","Data":"456cd86c318402eda2d1744b92485866ec0c4010e2e56db43913f9d8704c1d68"}
Sep 16 17:27:49.451386 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.451351    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fcxsr"
Sep 16 17:27:49.452844 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.452808    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n88th" event={"ID":"d87112b0-92d4-41bc-985f-572459ec8c6e","Type":"ContainerStarted","Data":"8a5bfae0bd8982269b6e2514db5c59b161cc26df0b9af983537cd5a09e6df0df"}
Sep 16 17:27:49.452968 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.452864    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n88th" event={"ID":"d87112b0-92d4-41bc-985f-572459ec8c6e","Type":"ContainerStarted","Data":"e5b0e15bdf277b7a1404e8fd1a49ad37d75b1f95a0ad74068b123a07c48764d5"}
Sep 16 17:27:49.456270 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.456249    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fcxsr"
Sep 16 17:27:49.475782 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.475734    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fcxsr" podStartSLOduration=2.3832475349999998 podStartE2EDuration="3.475717537s" totalImagesPullingTime="1.092470002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:49.474819893 +0000 UTC m=+198.263442438" watchObservedRunningTime="2026-09-16 17:27:49.475717537 +0000 UTC m=+198.264340070"
Sep 16 17:27:49.607149 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.607115    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:49.700022 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.699991    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"]
Sep 16 17:27:49.701737 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.701658    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:49.706189 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.705990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"
Sep 16 17:27:49.710818 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.710727    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 17:27:49.710965 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.710939    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 17:27:49.711028 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.710991    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ff5lb\""
Sep 16 17:27:49.711081 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.710731    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 17:27:49.711150 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.710944    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 17:27:49.711350 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.711330    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 17:27:49.723501 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.723475    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"]
Sep 16 17:27:49.816437 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.816394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p8th\" (UniqueName: \"kubernetes.io/projected/4c89cf2f-295e-4ac0-9c7b-e4ad9167a496-kube-api-access-5p8th\") pod \"prometheus-operator-68bcd7f987-jwp6h\" (UID: \"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"
Sep 16 17:27:49.816603 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.816444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c89cf2f-295e-4ac0-9c7b-e4ad9167a496-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-jwp6h\" (UID: \"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"
Sep 16 17:27:49.816603 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.816471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c89cf2f-295e-4ac0-9c7b-e4ad9167a496-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-jwp6h\" (UID: \"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"
Sep 16 17:27:49.816603 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.816576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c89cf2f-295e-4ac0-9c7b-e4ad9167a496-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-jwp6h\" (UID: \"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"
Sep 16 17:27:49.917858 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.917831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c89cf2f-295e-4ac0-9c7b-e4ad9167a496-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-jwp6h\" (UID: \"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"
Sep 16 17:27:49.918018 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.917870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c89cf2f-295e-4ac0-9c7b-e4ad9167a496-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-jwp6h\" (UID: \"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"
Sep 16 17:27:49.918018 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.917936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c89cf2f-295e-4ac0-9c7b-e4ad9167a496-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-jwp6h\" (UID: \"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"
Sep 16 17:27:49.918018 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:49.917991    2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 16 17:27:49.918174 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.918027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8th\" (UniqueName: \"kubernetes.io/projected/4c89cf2f-295e-4ac0-9c7b-e4ad9167a496-kube-api-access-5p8th\") pod \"prometheus-operator-68bcd7f987-jwp6h\" (UID: \"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"
Sep 16 17:27:49.918174 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:49.918092    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c89cf2f-295e-4ac0-9c7b-e4ad9167a496-prometheus-operator-tls podName:4c89cf2f-295e-4ac0-9c7b-e4ad9167a496 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:50.418072512 +0000 UTC m=+199.206695043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/4c89cf2f-295e-4ac0-9c7b-e4ad9167a496-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-jwp6h" (UID: "4c89cf2f-295e-4ac0-9c7b-e4ad9167a496") : secret "prometheus-operator-tls" not found
Sep 16 17:27:49.918959 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.918936    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c89cf2f-295e-4ac0-9c7b-e4ad9167a496-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-jwp6h\" (UID: \"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"
Sep 16 17:27:49.920772 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.920740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c89cf2f-295e-4ac0-9c7b-e4ad9167a496-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-jwp6h\" (UID: \"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"
Sep 16 17:27:49.927780 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:49.927754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8th\" (UniqueName: \"kubernetes.io/projected/4c89cf2f-295e-4ac0-9c7b-e4ad9167a496-kube-api-access-5p8th\") pod \"prometheus-operator-68bcd7f987-jwp6h\" (UID: \"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"
Sep 16 17:27:50.163008 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:50.162970    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:50.422466 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:50.422381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c89cf2f-295e-4ac0-9c7b-e4ad9167a496-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-jwp6h\" (UID: \"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"
Sep 16 17:27:50.425259 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:50.425229    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4c89cf2f-295e-4ac0-9c7b-e4ad9167a496-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-jwp6h\" (UID: \"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"
Sep 16 17:27:50.459345 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:50.459294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n88th" event={"ID":"d87112b0-92d4-41bc-985f-572459ec8c6e","Type":"ContainerStarted","Data":"434f23034c482fd78d3e10f0f0bacf89eb55da30d92cccc84b2610413f9cc06f"}
Sep 16 17:27:50.617990 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:50.617957    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"
Sep 16 17:27:50.913542 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:50.913456    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h"]
Sep 16 17:27:51.138845 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:27:51.138810    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c89cf2f_295e_4ac0_9c7b_e4ad9167a496.slice/crio-99cf56b0b7d52215059add819ad474a6c88cd7c39c8b1e24017c37328caa6109 WatchSource:0}: Error finding container 99cf56b0b7d52215059add819ad474a6c88cd7c39c8b1e24017c37328caa6109: Status 404 returned error can't find the container with id 99cf56b0b7d52215059add819ad474a6c88cd7c39c8b1e24017c37328caa6109
Sep 16 17:27:51.464287 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:51.464193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n88th" event={"ID":"d87112b0-92d4-41bc-985f-572459ec8c6e","Type":"ContainerStarted","Data":"85aec018e7194c125c8b6d2e654e0e158a7eb777e0f3f88f9552b7eecac7e02f"}
Sep 16 17:27:51.465471 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:51.465448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h" event={"ID":"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496","Type":"ContainerStarted","Data":"99cf56b0b7d52215059add819ad474a6c88cd7c39c8b1e24017c37328caa6109"}
Sep 16 17:27:51.486928 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:51.486855    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n88th" podStartSLOduration=33.620822988 podStartE2EDuration="35.486840905s" totalImagesPullingTime="1.866017917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:27:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:51.485204577 +0000 UTC m=+200.273827110" watchObservedRunningTime="2026-09-16 17:27:51.486840905 +0000 UTC m=+200.275463437"
Sep 16 17:27:52.470626 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:52.470533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h" event={"ID":"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496","Type":"ContainerStarted","Data":"01c7b88c89b02e38a1a5b510ec0219c380e5d08270280903529e194bd5660a43"}
Sep 16 17:27:52.470626 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:52.470588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h" event={"ID":"4c89cf2f-295e-4ac0-9c7b-e4ad9167a496","Type":"ContainerStarted","Data":"ffc847c1ee4668885f52ef17faec2156411d6c040c39ecbbb46ab09192744b80"}
Sep 16 17:27:52.475299 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:52.472149    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:52.491667 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:52.491616    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jwp6h" podStartSLOduration=2.453040266 podStartE2EDuration="3.491601597s" totalImagesPullingTime="1.038561331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:52.490970686 +0000 UTC m=+201.279593230" watchObservedRunningTime="2026-09-16 17:27:52.491601597 +0000 UTC m=+201.280224128"
Sep 16 17:27:53.049546 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:53.049508    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:53.110691 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:53.110658    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:53.762004 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:53.761971    2581 scope.go:117] "RemoveContainer" containerID="aa4316cd5f7e45b37627defd614f493ec4944527758776c9c198fff1c47be2a2"
Sep 16 17:27:53.762390 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:53.762140    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-xg8vh_openshift-console-operator(9e3a6183-641d-4047-9b87-6b79f0870b33)\"" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh" podUID="9e3a6183-641d-4047-9b87-6b79f0870b33"
Sep 16 17:27:53.787562 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:53.787525    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:54.076136 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.076103    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:54.182829 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.182793    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:54.198957 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.198910    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9cw8d"]
Sep 16 17:27:54.202752 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.202722    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"]
Sep 16 17:27:54.202912 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.202857    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.206826 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.206794    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"]
Sep 16 17:27:54.206962 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.206948    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.210226 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.210207    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"
Sep 16 17:27:54.211652 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.211083    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 17:27:54.215004 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.214981    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-z27np\""
Sep 16 17:27:54.215157 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.215142    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 17:27:54.215368 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.215204    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 17:27:54.215485 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.215244    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 17:27:54.215935 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.215912    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nv8hc\""
Sep 16 17:27:54.216030 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.215956    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 17:27:54.217075 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.217058    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-r5zdf\""
Sep 16 17:27:54.217191 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.217173    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 17:27:54.217328 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.217137    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 17:27:54.217492 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.217058    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 17:27:54.228562 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.228535    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"]
Sep 16 17:27:54.231327 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.231305    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"]
Sep 16 17:27:54.325485 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.325450    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:54.354744 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.354660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c451c7c3-281d-421b-b30a-0fb9eab2158b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-dmqg7\" (UID: \"c451c7c3-281d-421b-b30a-0fb9eab2158b\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"
Sep 16 17:27:54.354744 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.354693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nhqt\" (UniqueName: \"kubernetes.io/projected/c451c7c3-281d-421b-b30a-0fb9eab2158b-kube-api-access-5nhqt\") pod \"openshift-state-metrics-67dcf8c979-dmqg7\" (UID: \"c451c7c3-281d-421b-b30a-0fb9eab2158b\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"
Sep 16 17:27:54.354744 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.354711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4qfm\" (UniqueName: \"kubernetes.io/projected/8c9a0a70-06a4-4307-bfa9-4693b933f008-kube-api-access-q4qfm\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.355005 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.354776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c451c7c3-281d-421b-b30a-0fb9eab2158b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-dmqg7\" (UID: \"c451c7c3-281d-421b-b30a-0fb9eab2158b\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"
Sep 16 17:27:54.355005 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.354813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c9a0a70-06a4-4307-bfa9-4693b933f008-node-exporter-tls\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.355005 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.354912    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/28ee14c3-a8e9-4421-81a1-56f713758016-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.355005 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.354952    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28ee14c3-a8e9-4421-81a1-56f713758016-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.355175 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.355010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c9a0a70-06a4-4307-bfa9-4693b933f008-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.355175 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.355028    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28ee14c3-a8e9-4421-81a1-56f713758016-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.355175 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.355044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c9a0a70-06a4-4307-bfa9-4693b933f008-node-exporter-textfile\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.355175 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.355058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glnd9\" (UniqueName: \"kubernetes.io/projected/28ee14c3-a8e9-4421-81a1-56f713758016-kube-api-access-glnd9\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.355175 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.355100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c451c7c3-281d-421b-b30a-0fb9eab2158b-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-dmqg7\" (UID: \"c451c7c3-281d-421b-b30a-0fb9eab2158b\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"
Sep 16 17:27:54.355175 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.355161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c9a0a70-06a4-4307-bfa9-4693b933f008-node-exporter-wtmp\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.355405 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.355200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c9a0a70-06a4-4307-bfa9-4693b933f008-sys\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.355405 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.355218    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c9a0a70-06a4-4307-bfa9-4693b933f008-metrics-client-ca\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.355405 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.355237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/28ee14c3-a8e9-4421-81a1-56f713758016-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.355405 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.355257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c9a0a70-06a4-4307-bfa9-4693b933f008-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.355405 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.355273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/28ee14c3-a8e9-4421-81a1-56f713758016-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.355405 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.355318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c9a0a70-06a4-4307-bfa9-4693b933f008-root\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.452566 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.452525    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:54.456591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.456566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c451c7c3-281d-421b-b30a-0fb9eab2158b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-dmqg7\" (UID: \"c451c7c3-281d-421b-b30a-0fb9eab2158b\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"
Sep 16 17:27:54.456658 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.456611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nhqt\" (UniqueName: \"kubernetes.io/projected/c451c7c3-281d-421b-b30a-0fb9eab2158b-kube-api-access-5nhqt\") pod \"openshift-state-metrics-67dcf8c979-dmqg7\" (UID: \"c451c7c3-281d-421b-b30a-0fb9eab2158b\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"
Sep 16 17:27:54.456658 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.456638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4qfm\" (UniqueName: \"kubernetes.io/projected/8c9a0a70-06a4-4307-bfa9-4693b933f008-kube-api-access-q4qfm\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.456752 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.456698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c451c7c3-281d-421b-b30a-0fb9eab2158b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-dmqg7\" (UID: \"c451c7c3-281d-421b-b30a-0fb9eab2158b\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"
Sep 16 17:27:54.456752 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.456728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c9a0a70-06a4-4307-bfa9-4693b933f008-node-exporter-tls\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.456841 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.456767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/28ee14c3-a8e9-4421-81a1-56f713758016-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.456841 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.456794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28ee14c3-a8e9-4421-81a1-56f713758016-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.457205 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457178    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/28ee14c3-a8e9-4421-81a1-56f713758016-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.457324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c9a0a70-06a4-4307-bfa9-4693b933f008-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.457324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28ee14c3-a8e9-4421-81a1-56f713758016-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.457324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c9a0a70-06a4-4307-bfa9-4693b933f008-node-exporter-textfile\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.457482 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnd9\" (UniqueName: \"kubernetes.io/projected/28ee14c3-a8e9-4421-81a1-56f713758016-kube-api-access-glnd9\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.457482 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c451c7c3-281d-421b-b30a-0fb9eab2158b-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-dmqg7\" (UID: \"c451c7c3-281d-421b-b30a-0fb9eab2158b\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"
Sep 16 17:27:54.457482 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c9a0a70-06a4-4307-bfa9-4693b933f008-node-exporter-wtmp\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.457482 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c9a0a70-06a4-4307-bfa9-4693b933f008-sys\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.457482 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c9a0a70-06a4-4307-bfa9-4693b933f008-metrics-client-ca\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.457482 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/28ee14c3-a8e9-4421-81a1-56f713758016-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.457759 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c9a0a70-06a4-4307-bfa9-4693b933f008-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.457759 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/28ee14c3-a8e9-4421-81a1-56f713758016-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.457759 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c9a0a70-06a4-4307-bfa9-4693b933f008-root\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.457759 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c9a0a70-06a4-4307-bfa9-4693b933f008-root\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.457759 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:54.457702    2581 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 16 17:27:54.458011 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:27:54.457777    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28ee14c3-a8e9-4421-81a1-56f713758016-kube-state-metrics-tls podName:28ee14c3-a8e9-4421-81a1-56f713758016 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:54.957758502 +0000 UTC m=+203.746381031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/28ee14c3-a8e9-4421-81a1-56f713758016-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-975rr" (UID: "28ee14c3-a8e9-4421-81a1-56f713758016") : secret "kube-state-metrics-tls" not found
Sep 16 17:27:54.458011 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457792    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c9a0a70-06a4-4307-bfa9-4693b933f008-sys\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.458011 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.457834    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c9a0a70-06a4-4307-bfa9-4693b933f008-node-exporter-wtmp\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.458369 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.458343    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c9a0a70-06a4-4307-bfa9-4693b933f008-metrics-client-ca\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.458481 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.458456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/28ee14c3-a8e9-4421-81a1-56f713758016-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.458592 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.458573    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c9a0a70-06a4-4307-bfa9-4693b933f008-node-exporter-textfile\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.458918 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.458879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28ee14c3-a8e9-4421-81a1-56f713758016-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.459116 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.459096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c9a0a70-06a4-4307-bfa9-4693b933f008-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.459905 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.459852    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28ee14c3-a8e9-4421-81a1-56f713758016-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.459988 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.459437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c451c7c3-281d-421b-b30a-0fb9eab2158b-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-dmqg7\" (UID: \"c451c7c3-281d-421b-b30a-0fb9eab2158b\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"
Sep 16 17:27:54.460190 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.460164    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c451c7c3-281d-421b-b30a-0fb9eab2158b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-dmqg7\" (UID: \"c451c7c3-281d-421b-b30a-0fb9eab2158b\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"
Sep 16 17:27:54.460425 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.460405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c451c7c3-281d-421b-b30a-0fb9eab2158b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-dmqg7\" (UID: \"c451c7c3-281d-421b-b30a-0fb9eab2158b\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"
Sep 16 17:27:54.460532 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.460510    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c9a0a70-06a4-4307-bfa9-4693b933f008-node-exporter-tls\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.461350 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.461331    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c9a0a70-06a4-4307-bfa9-4693b933f008-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.468775 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.468754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nhqt\" (UniqueName: \"kubernetes.io/projected/c451c7c3-281d-421b-b30a-0fb9eab2158b-kube-api-access-5nhqt\") pod \"openshift-state-metrics-67dcf8c979-dmqg7\" (UID: \"c451c7c3-281d-421b-b30a-0fb9eab2158b\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"
Sep 16 17:27:54.477145 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.477119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4qfm\" (UniqueName: \"kubernetes.io/projected/8c9a0a70-06a4-4307-bfa9-4693b933f008-kube-api-access-q4qfm\") pod \"node-exporter-9cw8d\" (UID: \"8c9a0a70-06a4-4307-bfa9-4693b933f008\") " pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.479475 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.479457    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glnd9\" (UniqueName: \"kubernetes.io/projected/28ee14c3-a8e9-4421-81a1-56f713758016-kube-api-access-glnd9\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.515423 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.515387    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9cw8d"
Sep 16 17:27:54.527983 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:27:54.527954    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c9a0a70_06a4_4307_bfa9_4693b933f008.slice/crio-388c9f60ac460c435a7e59832d5273a5ccce589b4c22e5bc72804d12b6e3d431 WatchSource:0}: Error finding container 388c9f60ac460c435a7e59832d5273a5ccce589b4c22e5bc72804d12b6e3d431: Status 404 returned error can't find the container with id 388c9f60ac460c435a7e59832d5273a5ccce589b4c22e5bc72804d12b6e3d431
Sep 16 17:27:54.529119 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.529097    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"
Sep 16 17:27:54.710081 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.709981    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:54.809179 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.809147    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:54.824345 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.824317    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7"]
Sep 16 17:27:54.826646 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:27:54.826621    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc451c7c3_281d_421b_b30a_0fb9eab2158b.slice/crio-92bab6cedd12d28fc8626d42eb61b9bfdc2974814175fedad0792812b19fac93 WatchSource:0}: Error finding container 92bab6cedd12d28fc8626d42eb61b9bfdc2974814175fedad0792812b19fac93: Status 404 returned error can't find the container with id 92bab6cedd12d28fc8626d42eb61b9bfdc2974814175fedad0792812b19fac93
Sep 16 17:27:54.962463 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.962401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/28ee14c3-a8e9-4421-81a1-56f713758016-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:54.965408 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:54.965339    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/28ee14c3-a8e9-4421-81a1-56f713758016-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-975rr\" (UID: \"28ee14c3-a8e9-4421-81a1-56f713758016\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:55.108984 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:55.108946    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:55.122865 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:55.122831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"
Sep 16 17:27:55.226310 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:55.226205    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:55.397149 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:55.397110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:55.487913 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:55.487850    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7" event={"ID":"c451c7c3-281d-421b-b30a-0fb9eab2158b","Type":"ContainerStarted","Data":"c71a7d1e00b2aacfddae66fc606cb70e4df7143ba7a2409bf56213cd5446fc3d"}
Sep 16 17:27:55.488023 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:55.487967    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7" event={"ID":"c451c7c3-281d-421b-b30a-0fb9eab2158b","Type":"ContainerStarted","Data":"cff00644848eccf3e7461313a28568e558cfc6090e6eef5fc5f7a32d9a9f1838"}
Sep 16 17:27:55.488023 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:55.487987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7" event={"ID":"c451c7c3-281d-421b-b30a-0fb9eab2158b","Type":"ContainerStarted","Data":"92bab6cedd12d28fc8626d42eb61b9bfdc2974814175fedad0792812b19fac93"}
Sep 16 17:27:55.489791 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:55.489768    2581 generic.go:356] "Generic (PLEG): container finished" podID="8c9a0a70-06a4-4307-bfa9-4693b933f008" containerID="428ccd8c08dbe4da1af9328fa0db17833d4d443e7c37959bd14c01eb7b30d659" exitCode=0
Sep 16 17:27:55.489875 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:55.489815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cw8d" event={"ID":"8c9a0a70-06a4-4307-bfa9-4693b933f008","Type":"ContainerDied","Data":"428ccd8c08dbe4da1af9328fa0db17833d4d443e7c37959bd14c01eb7b30d659"}
Sep 16 17:27:55.489875 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:55.489851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cw8d" event={"ID":"8c9a0a70-06a4-4307-bfa9-4693b933f008","Type":"ContainerStarted","Data":"388c9f60ac460c435a7e59832d5273a5ccce589b4c22e5bc72804d12b6e3d431"}
Sep 16 17:27:55.514245 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:55.514216    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:55.574187 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:55.574162    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-975rr"]
Sep 16 17:27:55.575969 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:27:55.575943    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ee14c3_a8e9_4421_81a1_56f713758016.slice/crio-40ca1bcb03591db1e1461db1f65af1104a4b50ce7768194a9e2f40f4d4e70ee1 WatchSource:0}: Error finding container 40ca1bcb03591db1e1461db1f65af1104a4b50ce7768194a9e2f40f4d4e70ee1: Status 404 returned error can't find the container with id 40ca1bcb03591db1e1461db1f65af1104a4b50ce7768194a9e2f40f4d4e70ee1
Sep 16 17:27:56.496236 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:56.496141    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr" event={"ID":"28ee14c3-a8e9-4421-81a1-56f713758016","Type":"ContainerStarted","Data":"40ca1bcb03591db1e1461db1f65af1104a4b50ce7768194a9e2f40f4d4e70ee1"}
Sep 16 17:27:56.498653 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:56.498616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7" event={"ID":"c451c7c3-281d-421b-b30a-0fb9eab2158b","Type":"ContainerStarted","Data":"7752b2993fedc55d5d69ae6b49837616d488dc0f31ac6042855e535d55df60ed"}
Sep 16 17:27:56.501041 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:56.501022    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cw8d" event={"ID":"8c9a0a70-06a4-4307-bfa9-4693b933f008","Type":"ContainerStarted","Data":"0a9c402bcacc7f8b7498be19995cb0813f5ffd4d3c5e2cad4d3217cb4a05c1b9"}
Sep 16 17:27:56.501141 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:56.501072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cw8d" event={"ID":"8c9a0a70-06a4-4307-bfa9-4693b933f008","Type":"ContainerStarted","Data":"cbadb5d5834681730a02277452e97a927ce40b3f161d0950be0de00a99da76ef"}
Sep 16 17:27:56.527000 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:56.526944    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dmqg7" podStartSLOduration=1.297894026 podStartE2EDuration="2.526921854s" totalImagesPullingTime="1.229027828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:56.525055068 +0000 UTC m=+205.313677601" watchObservedRunningTime="2026-09-16 17:27:56.526921854 +0000 UTC m=+205.315544384"
Sep 16 17:27:56.553703 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:56.553640    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9cw8d" podStartSLOduration=1.67412685 podStartE2EDuration="2.553622385s" totalImagesPullingTime="808.702001ms" totalInitContainerRuntime="70.793534ms" isStatefulPod=true podCreationTimestamp="2026-09-16 17:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:56.553209714 +0000 UTC m=+205.341832248" watchObservedRunningTime="2026-09-16 17:27:56.553622385 +0000 UTC m=+205.342244917"
Sep 16 17:27:57.113015 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:57.112953    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:57.222040 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:57.222002    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:57.318298 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:57.318260    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:57.416181 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:57.416092    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:57.506530 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:57.506500    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr" event={"ID":"28ee14c3-a8e9-4421-81a1-56f713758016","Type":"ContainerStarted","Data":"395fe0dce99d0617cc3bd996513ec00305ee07ff6ad553e44b4acbedfd8cf6bd"}
Sep 16 17:27:57.507011 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:57.506562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr" event={"ID":"28ee14c3-a8e9-4421-81a1-56f713758016","Type":"ContainerStarted","Data":"1920892bc06e5443cea1c65d080207b786143492519e4de9d8a8310d20d91865"}
Sep 16 17:27:57.507011 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:57.506591    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr" event={"ID":"28ee14c3-a8e9-4421-81a1-56f713758016","Type":"ContainerStarted","Data":"836f3bfc9564ed4804abc543581b97ff8a1617478e83abf3e2d2a022dfc185e4"}
Sep 16 17:27:57.549173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:57.549120    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-975rr" podStartSLOduration=2.4267090700000002 podStartE2EDuration="3.549101704s" totalImagesPullingTime="1.122392634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:57.548672703 +0000 UTC m=+206.337295236" watchObservedRunningTime="2026-09-16 17:27:57.549101704 +0000 UTC m=+206.337724237"
Sep 16 17:27:58.876876 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:58.876839    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:59.152279 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:27:59.152189    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:00.477594 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.477486    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:00.479795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.479769    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:28:00.484320 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.484300    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.489342 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.489017    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 17:28:00.489342 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.489113    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 17:28:00.489342 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.489145    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 17:28:00.489342 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.489175    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 17:28:00.489342 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.489201    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 17:28:00.489342 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.489242    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 17:28:00.489342 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.489308    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 17:28:00.490191 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.490172    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-b9p8j\""
Sep 16 17:28:00.490511 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.490490    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 17:28:00.490667 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.490647    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 17:28:00.490744 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.490672    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 17:28:00.490744 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.490709    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 17:28:00.490851 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.490655    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 17:28:00.491013 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.490997    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-537248iqg2jv2\""
Sep 16 17:28:00.491148 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.491127    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 17:28:00.491247 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.491084    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 17:28:00.495319 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.495290    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 17:28:00.509240 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.509213    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:28:00.610494 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.610461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.610494 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.610500    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.610704 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.610530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.610704 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.610551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-web-config\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.610704 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.610573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.610704 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.610627    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.610704 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.610684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.610930 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.610724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzcbw\" (UniqueName: \"kubernetes.io/projected/7983b215-4d22-4c1c-9254-24464938f77e-kube-api-access-pzcbw\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.610930 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.610742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.610930 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.610762    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.610930 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.610856    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.610930 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.610906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.610930 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.610930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7983b215-4d22-4c1c-9254-24464938f77e-config-out\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.611171 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.610991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-config\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.611171 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.611015    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.611171 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.611039    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.611171 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.611061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.611171 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.611090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7983b215-4d22-4c1c-9254-24464938f77e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.611171 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.611111    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.611171 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.611145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.711739 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.711700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.711949 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.711753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzcbw\" (UniqueName: \"kubernetes.io/projected/7983b215-4d22-4c1c-9254-24464938f77e-kube-api-access-pzcbw\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.711949 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.711772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.711949 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.711791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.711949 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.711819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.711949 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.711842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.711949 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.711872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7983b215-4d22-4c1c-9254-24464938f77e-config-out\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.711949 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.711935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-config\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.712324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.711960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.712324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.711985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.712324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.712015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.712324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.712049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7983b215-4d22-4c1c-9254-24464938f77e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.712324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.712071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.712324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.712105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.712324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.712168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.712324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.712191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.712324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.712221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.712324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.712243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-web-config\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.712324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.712269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.712324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.712296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.712855 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.712600    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.712987 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.712963    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.713045 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.713005    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.713836 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.713813    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.715056 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.715034    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.715647 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.715562    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.718633 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.716007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.718633 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.716383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.718633 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.716752    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-config\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.718633 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.718589    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.718880 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.718678    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-web-config\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.719144 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.719078    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7983b215-4d22-4c1c-9254-24464938f77e-config-out\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.719408 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.719378    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.719508 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.719477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.719632 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.719525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.720314 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.720280    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.721169 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.721146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7983b215-4d22-4c1c-9254-24464938f77e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.721859 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.721837    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.722203 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.722184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzcbw\" (UniqueName: \"kubernetes.io/projected/7983b215-4d22-4c1c-9254-24464938f77e-kube-api-access-pzcbw\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.722247 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.722229    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:00.796712 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:00.796670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:01.095253 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:28:01.095213    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7983b215_4d22_4c1c_9254_24464938f77e.slice/crio-28e93dcaf2f2c23d5824e28a5f240ba9fc14c6731ee6b184e05c40ccf88b9f67 WatchSource:0}: Error finding container 28e93dcaf2f2c23d5824e28a5f240ba9fc14c6731ee6b184e05c40ccf88b9f67: Status 404 returned error can't find the container with id 28e93dcaf2f2c23d5824e28a5f240ba9fc14c6731ee6b184e05c40ccf88b9f67
Sep 16 17:28:01.095637 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:01.095614    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:28:01.523450 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:01.523411    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7983b215-4d22-4c1c-9254-24464938f77e","Type":"ContainerStarted","Data":"28e93dcaf2f2c23d5824e28a5f240ba9fc14c6731ee6b184e05c40ccf88b9f67"}
Sep 16 17:28:02.338777 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:02.338696    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:02.441305 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:02.441260    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:02.529261 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:02.529223    2581 generic.go:356] "Generic (PLEG): container finished" podID="7983b215-4d22-4c1c-9254-24464938f77e" containerID="013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8" exitCode=0
Sep 16 17:28:02.529645 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:02.529296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7983b215-4d22-4c1c-9254-24464938f77e","Type":"ContainerDied","Data":"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8"}
Sep 16 17:28:04.399169 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:04.399131    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:04.455312 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:04.455270    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:04.761991 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:04.761909    2581 scope.go:117] "RemoveContainer" containerID="aa4316cd5f7e45b37627defd614f493ec4944527758776c9c198fff1c47be2a2"
Sep 16 17:28:05.544239 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:05.544219    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:28:05.544615 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:05.544591    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh" event={"ID":"9e3a6183-641d-4047-9b87-6b79f0870b33","Type":"ContainerStarted","Data":"1ed3a084d6776ca45ef7f1cd2b0fc1228a4af829a4ed5f021d856f8f7146267e"}
Sep 16 17:28:05.545104 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:05.545083    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:28:05.546979 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:05.546854    2581 patch_prober.go:28] interesting pod/console-operator-696884fb9c-xg8vh container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.9:8443/readyz\": dial tcp 10.132.0.9:8443: connect: connection refused" start-of-body=
Sep 16 17:28:05.547079 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:05.546965    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh" podUID="9e3a6183-641d-4047-9b87-6b79f0870b33" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.9:8443/readyz\": dial tcp 10.132.0.9:8443: connect: connection refused"
Sep 16 17:28:05.568405 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:05.568356    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh" podStartSLOduration=52.714026898 podStartE2EDuration="54.568341455s" totalImagesPullingTime="1.854314557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:28:05.566951909 +0000 UTC m=+214.355574444" watchObservedRunningTime="2026-09-16 17:28:05.568341455 +0000 UTC m=+214.356963988"
Sep 16 17:28:06.553973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:06.553935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7983b215-4d22-4c1c-9254-24464938f77e","Type":"ContainerStarted","Data":"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"}
Sep 16 17:28:06.554398 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:06.554021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7983b215-4d22-4c1c-9254-24464938f77e","Type":"ContainerStarted","Data":"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"}
Sep 16 17:28:06.613591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:06.613561    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-xg8vh"
Sep 16 17:28:06.830144 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:06.829824    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:06.921714 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:06.921677    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:06.980562 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:06.980502    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:07.286258 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.286223    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b89f8fc75-6zrbw"]
Sep 16 17:28:07.286491 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:28:07.286473    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw" podUID="d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a"
Sep 16 17:28:07.320328 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.320291    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:07.406123 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.406076    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:07.508335 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.508302    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:07.562114 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.562082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7983b215-4d22-4c1c-9254-24464938f77e","Type":"ContainerStarted","Data":"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"}
Sep 16 17:28:07.562535 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.562092    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:28:07.562535 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.562168    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7983b215-4d22-4c1c-9254-24464938f77e","Type":"ContainerStarted","Data":"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"}
Sep 16 17:28:07.562535 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.562205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7983b215-4d22-4c1c-9254-24464938f77e","Type":"ContainerStarted","Data":"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"}
Sep 16 17:28:07.562535 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.562231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7983b215-4d22-4c1c-9254-24464938f77e","Type":"ContainerStarted","Data":"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"}
Sep 16 17:28:07.569281 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.569260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:28:07.582013 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.581992    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-trusted-ca\") pod \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") "
Sep 16 17:28:07.582128 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.582026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzwgn\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-kube-api-access-nzwgn\") pod \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") "
Sep 16 17:28:07.582128 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.582046    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-image-registry-private-configuration\") pod \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") "
Sep 16 17:28:07.582208 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.582182    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-certificates\") pod \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") "
Sep 16 17:28:07.582250 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.582211    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-ca-trust-extracted\") pod \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") "
Sep 16 17:28:07.582289 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.582269    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-bound-sa-token\") pod \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") "
Sep 16 17:28:07.582328 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.582309    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-installation-pull-secrets\") pod \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\" (UID: \"d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a\") "
Sep 16 17:28:07.582481 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.582449    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:28:07.582481 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.582467    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:07.582481 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.582476    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:07.584192 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.583280    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-trusted-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:28:07.584192 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.583308    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-certificates\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:28:07.584192 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.583323    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-ca-trust-extracted\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:28:07.584973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.584947    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:07.585144 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.585119    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:28:07.585192 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.585149    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:07.585224 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.585195    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-kube-api-access-nzwgn" (OuterVolumeSpecName: "kube-api-access-nzwgn") pod "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a" (UID: "d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a"). InnerVolumeSpecName "kube-api-access-nzwgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:28:07.594346 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.594313    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:07.631881 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.631820    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.025887863 podStartE2EDuration="7.631798779s" totalImagesPullingTime="5.589957138s" totalInitContainerRuntime="15.953778ms" isStatefulPod=false podCreationTimestamp="2026-09-16 17:28:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:28:07.629453663 +0000 UTC m=+216.418076196" watchObservedRunningTime="2026-09-16 17:28:07.631798779 +0000 UTC m=+216.420421310"
Sep 16 17:28:07.681413 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.681371    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:07.684470 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.684451    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-installation-pull-secrets\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:28:07.684543 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.684475    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzwgn\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-kube-api-access-nzwgn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:28:07.684543 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.684488    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-image-registry-private-configuration\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:28:07.684543 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.684497    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-bound-sa-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:28:07.863718 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.863635    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:07.930435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:07.930405    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:08.279413 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:08.279378    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:08.494026 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:08.493790    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:08.566465 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:08.566373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b89f8fc75-6zrbw"
Sep 16 17:28:08.606998 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:08.606967    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b89f8fc75-6zrbw"]
Sep 16 17:28:08.611308 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:08.611276    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b89f8fc75-6zrbw"]
Sep 16 17:28:08.693371 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:08.693336    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a-registry-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:28:08.884029 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:08.883930    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:08.985401 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:08.985359    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:09.764104 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:09.764076    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:09.766400 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:09.766378    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a" path="/var/lib/kubelet/pods/d4f49dcd-0323-4d5f-b2a5-a5bdbbcd131a/volumes"
Sep 16 17:28:09.826724 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:09.826674    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:09.991439 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:09.991405    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:10.092575 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:10.092539    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:10.162806 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:10.162760    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:10.797295 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:10.797257    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:16.174426 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:16.174385    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:16.277638 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:16.277599    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:16.501498 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:16.501406    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:16.558267 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:16.558234    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:21.708638 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:21.708608    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:21.770751 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:21.770708    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:25.111034 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:25.110990    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:25.172066 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:25.172031    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:28.507411 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:28.507368    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:28.564540 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:28.564508    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:29.419564 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:29.419529    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:29.473182 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:29.473151    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:31.655377 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:31.655292    2581 generic.go:356] "Generic (PLEG): container finished" podID="9395826b-e1df-4879-aa43-86e2af895fe8" containerID="e32c5ae44bdb25b482c9d7ad16bc217891e7986863599b4bea5640d787512e98" exitCode=0
Sep 16 17:28:31.655819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:31.655375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4" event={"ID":"9395826b-e1df-4879-aa43-86e2af895fe8","Type":"ContainerDied","Data":"e32c5ae44bdb25b482c9d7ad16bc217891e7986863599b4bea5640d787512e98"}
Sep 16 17:28:31.655819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:31.655751    2581 scope.go:117] "RemoveContainer" containerID="e32c5ae44bdb25b482c9d7ad16bc217891e7986863599b4bea5640d787512e98"
Sep 16 17:28:32.666708 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:32.666674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xsmh4" event={"ID":"9395826b-e1df-4879-aa43-86e2af895fe8","Type":"ContainerStarted","Data":"8f0c26b9dae56aeef2e82f0efc4b042c9b8f6b2733a88b9c81c67eab2f4243ca"}
Sep 16 17:28:34.512394 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:34.512353    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:34.564953 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:34.564921    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:35.119446 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:35.119411    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:35.220799 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:35.220758    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:35.562806 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:35.562767    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:36.741845 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:36.741809    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:36.796150 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:36.796120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:43.489651 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:43.489606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs\") pod \"network-metrics-daemon-jhf5t\" (UID: \"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8\") " pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:28:43.492062 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:43.492039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8-metrics-certs\") pod \"network-metrics-daemon-jhf5t\" (UID: \"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8\") " pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:28:43.665299 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:43.665266    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vhb2d\""
Sep 16 17:28:43.673128 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:43.673102    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jhf5t"
Sep 16 17:28:43.944168 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:43.944106    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jhf5t"]
Sep 16 17:28:43.946209 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:28:43.946179    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc6fd3c_7b7f_421f_8318_8d0ef38c4dc8.slice/crio-acdd6a57e9ecc4226d2a3787e339960867a134d56b6ec6514f5397bedfa56faa WatchSource:0}: Error finding container acdd6a57e9ecc4226d2a3787e339960867a134d56b6ec6514f5397bedfa56faa: Status 404 returned error can't find the container with id acdd6a57e9ecc4226d2a3787e339960867a134d56b6ec6514f5397bedfa56faa
Sep 16 17:28:44.410766 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:44.410722    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:44.466031 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:44.465988    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:44.718114 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:44.718030    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jhf5t" event={"ID":"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8","Type":"ContainerStarted","Data":"acdd6a57e9ecc4226d2a3787e339960867a134d56b6ec6514f5397bedfa56faa"}
Sep 16 17:28:45.723265 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:45.723228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jhf5t" event={"ID":"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8","Type":"ContainerStarted","Data":"166348b6868519b0a2a76b1d7ef778403e7bf57f374a33f0252b4b83baee7314"}
Sep 16 17:28:45.723633 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:45.723276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jhf5t" event={"ID":"fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8","Type":"ContainerStarted","Data":"2a6c2bb28d228864d2918977c8c3eb247032baf4f78857015644954700885bc0"}
Sep 16 17:28:45.741617 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:45.741568    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jhf5t" podStartSLOduration=252.783282981 podStartE2EDuration="4m13.74155085s" totalImagesPullingTime="958.267869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:28:45.739933597 +0000 UTC m=+254.528556130" watchObservedRunningTime="2026-09-16 17:28:45.74155085 +0000 UTC m=+254.530173382"
Sep 16 17:28:45.849205 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:45.849173    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:45.915426 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:45.915392    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:45.966235 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:45.966207    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:51.065810 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:51.065754    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:59.425387 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:59.425302    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:59.496218 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:28:59.496183    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:00.797412 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:00.797376    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:00.816296 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:00.816272    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:01.801217 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:01.801190    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:05.657293 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:05.657262    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:05.758094 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:05.758059    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:13.292477 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:29:13.292434    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-klh2p" podUID="a9f7a40a-f3c9-4e73-940e-666151905ce4"
Sep 16 17:29:13.382626 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:13.382592    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:13.488781 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:13.488744    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:13.831945 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:13.831885    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-klh2p"
Sep 16 17:29:14.158227 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:14.158132    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:16.770904 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:16.770867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:29:16.773307 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:16.773290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9f7a40a-f3c9-4e73-940e-666151905ce4-metrics-tls\") pod \"dns-default-klh2p\" (UID: \"a9f7a40a-f3c9-4e73-940e-666151905ce4\") " pod="openshift-dns/dns-default-klh2p"
Sep 16 17:29:16.835311 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:16.835282    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwbrz\""
Sep 16 17:29:16.842756 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:16.842734    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-klh2p"
Sep 16 17:29:17.116715 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:17.116645    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-klh2p"]
Sep 16 17:29:17.120453 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:29:17.120416    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f7a40a_f3c9_4e73_940e_666151905ce4.slice/crio-eb6045af9ddfa6ea195c3fadafc288139544a1c24f12c340a29ab8094c965fb4 WatchSource:0}: Error finding container eb6045af9ddfa6ea195c3fadafc288139544a1c24f12c340a29ab8094c965fb4: Status 404 returned error can't find the container with id eb6045af9ddfa6ea195c3fadafc288139544a1c24f12c340a29ab8094c965fb4
Sep 16 17:29:17.853540 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:17.853503    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-klh2p" event={"ID":"a9f7a40a-f3c9-4e73-940e-666151905ce4","Type":"ContainerStarted","Data":"eb6045af9ddfa6ea195c3fadafc288139544a1c24f12c340a29ab8094c965fb4"}
Sep 16 17:29:18.908457 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:18.907673    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:29:18.908457 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:18.908060    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="prometheus" containerID="cri-o://8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc" gracePeriod=600
Sep 16 17:29:18.908457 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:18.908097    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401" gracePeriod=600
Sep 16 17:29:18.908457 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:18.908242    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="kube-rbac-proxy" containerID="cri-o://1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15" gracePeriod=600
Sep 16 17:29:18.908457 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:18.908323    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="kube-rbac-proxy-web" containerID="cri-o://87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed" gracePeriod=600
Sep 16 17:29:18.908457 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:18.908389    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="thanos-sidecar" containerID="cri-o://902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0" gracePeriod=600
Sep 16 17:29:18.908457 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:18.908457    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="config-reloader" containerID="cri-o://c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0" gracePeriod=600
Sep 16 17:29:19.167004 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.166978    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.292411 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292379    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-web-config\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292411 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292415    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-metrics-client-ca\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292619 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292449    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-db\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292619 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292464    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzcbw\" (UniqueName: \"kubernetes.io/projected/7983b215-4d22-4c1c-9254-24464938f77e-kube-api-access-pzcbw\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292619 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292481    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-1\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292619 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292511    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292619 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292550    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-tls\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292619 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292583    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-2\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292619 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292611    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-kubelet-serving-ca-bundle\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292649    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-trusted-ca-bundle\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292685    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-config\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-metrics-client-certs\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292744    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-thanos-prometheus-http-client-file\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292770    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-0\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292794    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-serving-certs-ca-bundle\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292800    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:29:19.292992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292810    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:29:19.292992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292819    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292882    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7983b215-4d22-4c1c-9254-24464938f77e-config-out\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292952    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-kube-rbac-proxy\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.292992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.292981    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7983b215-4d22-4c1c-9254-24464938f77e-tls-assets\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.293636 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.293010    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-grpc-tls\") pod \"7983b215-4d22-4c1c-9254-24464938f77e\" (UID: \"7983b215-4d22-4c1c-9254-24464938f77e\") "
Sep 16 17:29:19.293636 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.293197    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:29:19.293636 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.293297    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.293636 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.293317    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.293636 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.293331    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-metrics-client-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.294077 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.293674    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:29:19.294077 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.294002    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:29:19.294497 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.294234    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:29:19.295498 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.295464    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7983b215-4d22-4c1c-9254-24464938f77e-kube-api-access-pzcbw" (OuterVolumeSpecName: "kube-api-access-pzcbw") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "kube-api-access-pzcbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:29:19.295694 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.295663    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-config" (OuterVolumeSpecName: "config") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:29:19.295796 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.295775    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:29:19.296077 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.295963    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:29:19.296325 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.296238    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:29:19.296819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.296748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:29:19.297247 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.297117    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:29:19.297398 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.297353    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:29:19.297699 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.297680    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:29:19.297813 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.297792    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:29:19.298017 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.298000    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:29:19.298278 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.298254    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7983b215-4d22-4c1c-9254-24464938f77e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:29:19.299132 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.299108    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7983b215-4d22-4c1c-9254-24464938f77e-config-out" (OuterVolumeSpecName: "config-out") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:29:19.306435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.306412    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-web-config" (OuterVolumeSpecName: "web-config") pod "7983b215-4d22-4c1c-9254-24464938f77e" (UID: "7983b215-4d22-4c1c-9254-24464938f77e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:29:19.393973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.393932    2581 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-grpc-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.393973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.393963    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-web-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.393973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.393971    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-db\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.393973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.393982    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzcbw\" (UniqueName: \"kubernetes.io/projected/7983b215-4d22-4c1c-9254-24464938f77e-kube-api-access-pzcbw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.394228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.393993    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.394228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.394002    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.394228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.394010    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.394228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.394022    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.394228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.394030    2581 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.394228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.394037    2581 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-metrics-client-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.394228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.394045    2581 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.394228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.394053    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.394228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.394061    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7983b215-4d22-4c1c-9254-24464938f77e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.394228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.394069    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.394228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.394078    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7983b215-4d22-4c1c-9254-24464938f77e-config-out\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.394228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.394086    2581 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7983b215-4d22-4c1c-9254-24464938f77e-secret-kube-rbac-proxy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.394228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.394093    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7983b215-4d22-4c1c-9254-24464938f77e-tls-assets\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:29:19.864429 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.864396    2581 generic.go:356] "Generic (PLEG): container finished" podID="7983b215-4d22-4c1c-9254-24464938f77e" containerID="2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401" exitCode=0
Sep 16 17:29:19.864429 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.864419    2581 generic.go:356] "Generic (PLEG): container finished" podID="7983b215-4d22-4c1c-9254-24464938f77e" containerID="1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15" exitCode=0
Sep 16 17:29:19.864429 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.864427    2581 generic.go:356] "Generic (PLEG): container finished" podID="7983b215-4d22-4c1c-9254-24464938f77e" containerID="87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed" exitCode=0
Sep 16 17:29:19.864429 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.864433    2581 generic.go:356] "Generic (PLEG): container finished" podID="7983b215-4d22-4c1c-9254-24464938f77e" containerID="902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0" exitCode=0
Sep 16 17:29:19.864429 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.864440    2581 generic.go:356] "Generic (PLEG): container finished" podID="7983b215-4d22-4c1c-9254-24464938f77e" containerID="c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0" exitCode=0
Sep 16 17:29:19.864797 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.864447    2581 generic.go:356] "Generic (PLEG): container finished" podID="7983b215-4d22-4c1c-9254-24464938f77e" containerID="8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc" exitCode=0
Sep 16 17:29:19.864797 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.864474    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.864797 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.864481    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7983b215-4d22-4c1c-9254-24464938f77e","Type":"ContainerDied","Data":"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"}
Sep 16 17:29:19.864797 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.864572    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7983b215-4d22-4c1c-9254-24464938f77e","Type":"ContainerDied","Data":"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"}
Sep 16 17:29:19.864797 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.864615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7983b215-4d22-4c1c-9254-24464938f77e","Type":"ContainerDied","Data":"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"}
Sep 16 17:29:19.864797 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.864637    2581 scope.go:117] "RemoveContainer" containerID="2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"
Sep 16 17:29:19.864797 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.864655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7983b215-4d22-4c1c-9254-24464938f77e","Type":"ContainerDied","Data":"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"}
Sep 16 17:29:19.864797 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.864696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7983b215-4d22-4c1c-9254-24464938f77e","Type":"ContainerDied","Data":"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"}
Sep 16 17:29:19.864797 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.864735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7983b215-4d22-4c1c-9254-24464938f77e","Type":"ContainerDied","Data":"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"}
Sep 16 17:29:19.864797 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.864772    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7983b215-4d22-4c1c-9254-24464938f77e","Type":"ContainerDied","Data":"28e93dcaf2f2c23d5824e28a5f240ba9fc14c6731ee6b184e05c40ccf88b9f67"}
Sep 16 17:29:19.866626 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.866603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-klh2p" event={"ID":"a9f7a40a-f3c9-4e73-940e-666151905ce4","Type":"ContainerStarted","Data":"2f78075baaee65cbd2b84e2eb86c8215327471252e4fbe431f1562bbfb3a47ba"}
Sep 16 17:29:19.866801 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.866647    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-klh2p" event={"ID":"a9f7a40a-f3c9-4e73-940e-666151905ce4","Type":"ContainerStarted","Data":"3a7f6dafab4630149857636dde6dbca6ccae3cb377e9874e616a3096ab2073cd"}
Sep 16 17:29:19.866801 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.866713    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-klh2p"
Sep 16 17:29:19.872828 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.872812    2581 scope.go:117] "RemoveContainer" containerID="1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"
Sep 16 17:29:19.879541 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.879524    2581 scope.go:117] "RemoveContainer" containerID="87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"
Sep 16 17:29:19.886039 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.886021    2581 scope.go:117] "RemoveContainer" containerID="902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"
Sep 16 17:29:19.890357 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.890311    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-klh2p" podStartSLOduration=252.217879617 podStartE2EDuration="4m13.890293389s" totalImagesPullingTime="1.672413772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:29:19.889092715 +0000 UTC m=+288.677715274" watchObservedRunningTime="2026-09-16 17:29:19.890293389 +0000 UTC m=+288.678915922"
Sep 16 17:29:19.893107 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.893091    2581 scope.go:117] "RemoveContainer" containerID="c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"
Sep 16 17:29:19.899728 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.899707    2581 scope.go:117] "RemoveContainer" containerID="8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"
Sep 16 17:29:19.904772 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.904752    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:29:19.907443 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.907418    2581 scope.go:117] "RemoveContainer" containerID="013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8"
Sep 16 17:29:19.909272 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.909254    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:29:19.913975 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.913960    2581 scope.go:117] "RemoveContainer" containerID="2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"
Sep 16 17:29:19.914276 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:29:19.914247    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401\": container with ID starting with 2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401 not found: ID does not exist" containerID="2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"
Sep 16 17:29:19.914354 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.914277    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"} err="failed to get container status \"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401\": rpc error: code = NotFound desc = could not find container \"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401\": container with ID starting with 2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401 not found: ID does not exist"
Sep 16 17:29:19.914354 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.914315    2581 scope.go:117] "RemoveContainer" containerID="1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"
Sep 16 17:29:19.914595 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:29:19.914578    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15\": container with ID starting with 1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15 not found: ID does not exist" containerID="1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"
Sep 16 17:29:19.914638 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.914600    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"} err="failed to get container status \"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15\": rpc error: code = NotFound desc = could not find container \"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15\": container with ID starting with 1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15 not found: ID does not exist"
Sep 16 17:29:19.914638 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.914616    2581 scope.go:117] "RemoveContainer" containerID="87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"
Sep 16 17:29:19.914830 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:29:19.914815    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed\": container with ID starting with 87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed not found: ID does not exist" containerID="87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"
Sep 16 17:29:19.914872 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.914835    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"} err="failed to get container status \"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed\": rpc error: code = NotFound desc = could not find container \"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed\": container with ID starting with 87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed not found: ID does not exist"
Sep 16 17:29:19.914872 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.914847    2581 scope.go:117] "RemoveContainer" containerID="902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"
Sep 16 17:29:19.915083 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:29:19.915058    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0\": container with ID starting with 902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0 not found: ID does not exist" containerID="902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"
Sep 16 17:29:19.915128 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.915091    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"} err="failed to get container status \"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0\": rpc error: code = NotFound desc = could not find container \"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0\": container with ID starting with 902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0 not found: ID does not exist"
Sep 16 17:29:19.915128 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.915109    2581 scope.go:117] "RemoveContainer" containerID="c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"
Sep 16 17:29:19.915360 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:29:19.915344    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0\": container with ID starting with c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0 not found: ID does not exist" containerID="c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"
Sep 16 17:29:19.915418 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.915367    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"} err="failed to get container status \"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0\": rpc error: code = NotFound desc = could not find container \"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0\": container with ID starting with c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0 not found: ID does not exist"
Sep 16 17:29:19.915418 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.915387    2581 scope.go:117] "RemoveContainer" containerID="8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"
Sep 16 17:29:19.915603 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:29:19.915588    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc\": container with ID starting with 8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc not found: ID does not exist" containerID="8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"
Sep 16 17:29:19.915639 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.915606    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"} err="failed to get container status \"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc\": rpc error: code = NotFound desc = could not find container \"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc\": container with ID starting with 8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc not found: ID does not exist"
Sep 16 17:29:19.915639 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.915618    2581 scope.go:117] "RemoveContainer" containerID="013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8"
Sep 16 17:29:19.915844 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:29:19.915827    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8\": container with ID starting with 013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8 not found: ID does not exist" containerID="013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8"
Sep 16 17:29:19.915918 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.915849    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8"} err="failed to get container status \"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8\": rpc error: code = NotFound desc = could not find container \"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8\": container with ID starting with 013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8 not found: ID does not exist"
Sep 16 17:29:19.915918 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.915863    2581 scope.go:117] "RemoveContainer" containerID="2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"
Sep 16 17:29:19.916093 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.916074    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"} err="failed to get container status \"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401\": rpc error: code = NotFound desc = could not find container \"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401\": container with ID starting with 2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401 not found: ID does not exist"
Sep 16 17:29:19.916152 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.916094    2581 scope.go:117] "RemoveContainer" containerID="1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"
Sep 16 17:29:19.916312 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.916296    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"} err="failed to get container status \"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15\": rpc error: code = NotFound desc = could not find container \"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15\": container with ID starting with 1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15 not found: ID does not exist"
Sep 16 17:29:19.916355 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.916312    2581 scope.go:117] "RemoveContainer" containerID="87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"
Sep 16 17:29:19.916528 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.916511    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"} err="failed to get container status \"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed\": rpc error: code = NotFound desc = could not find container \"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed\": container with ID starting with 87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed not found: ID does not exist"
Sep 16 17:29:19.916595 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.916529    2581 scope.go:117] "RemoveContainer" containerID="902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"
Sep 16 17:29:19.916741 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.916724    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"} err="failed to get container status \"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0\": rpc error: code = NotFound desc = could not find container \"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0\": container with ID starting with 902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0 not found: ID does not exist"
Sep 16 17:29:19.916785 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.916742    2581 scope.go:117] "RemoveContainer" containerID="c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"
Sep 16 17:29:19.916956 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.916941    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"} err="failed to get container status \"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0\": rpc error: code = NotFound desc = could not find container \"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0\": container with ID starting with c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0 not found: ID does not exist"
Sep 16 17:29:19.917010 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.916958    2581 scope.go:117] "RemoveContainer" containerID="8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"
Sep 16 17:29:19.917168 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.917151    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"} err="failed to get container status \"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc\": rpc error: code = NotFound desc = could not find container \"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc\": container with ID starting with 8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc not found: ID does not exist"
Sep 16 17:29:19.917232 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.917169    2581 scope.go:117] "RemoveContainer" containerID="013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8"
Sep 16 17:29:19.917379 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.917361    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8"} err="failed to get container status \"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8\": rpc error: code = NotFound desc = could not find container \"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8\": container with ID starting with 013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8 not found: ID does not exist"
Sep 16 17:29:19.917419 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.917380    2581 scope.go:117] "RemoveContainer" containerID="2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"
Sep 16 17:29:19.917591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.917575    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"} err="failed to get container status \"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401\": rpc error: code = NotFound desc = could not find container \"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401\": container with ID starting with 2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401 not found: ID does not exist"
Sep 16 17:29:19.917637 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.917591    2581 scope.go:117] "RemoveContainer" containerID="1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"
Sep 16 17:29:19.917804 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.917788    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"} err="failed to get container status \"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15\": rpc error: code = NotFound desc = could not find container \"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15\": container with ID starting with 1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15 not found: ID does not exist"
Sep 16 17:29:19.917868 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.917806    2581 scope.go:117] "RemoveContainer" containerID="87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"
Sep 16 17:29:19.918016 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.917998    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"} err="failed to get container status \"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed\": rpc error: code = NotFound desc = could not find container \"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed\": container with ID starting with 87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed not found: ID does not exist"
Sep 16 17:29:19.918058 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.918017    2581 scope.go:117] "RemoveContainer" containerID="902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"
Sep 16 17:29:19.918225 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.918209    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"} err="failed to get container status \"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0\": rpc error: code = NotFound desc = could not find container \"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0\": container with ID starting with 902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0 not found: ID does not exist"
Sep 16 17:29:19.918270 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.918239    2581 scope.go:117] "RemoveContainer" containerID="c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"
Sep 16 17:29:19.918445 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.918427    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"} err="failed to get container status \"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0\": rpc error: code = NotFound desc = could not find container \"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0\": container with ID starting with c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0 not found: ID does not exist"
Sep 16 17:29:19.918509 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.918446    2581 scope.go:117] "RemoveContainer" containerID="8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"
Sep 16 17:29:19.918679 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.918664    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"} err="failed to get container status \"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc\": rpc error: code = NotFound desc = could not find container \"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc\": container with ID starting with 8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc not found: ID does not exist"
Sep 16 17:29:19.918722 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.918679    2581 scope.go:117] "RemoveContainer" containerID="013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8"
Sep 16 17:29:19.918919 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.918880    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8"} err="failed to get container status \"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8\": rpc error: code = NotFound desc = could not find container \"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8\": container with ID starting with 013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8 not found: ID does not exist"
Sep 16 17:29:19.918919 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.918919    2581 scope.go:117] "RemoveContainer" containerID="2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"
Sep 16 17:29:19.919127 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.919110    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"} err="failed to get container status \"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401\": rpc error: code = NotFound desc = could not find container \"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401\": container with ID starting with 2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401 not found: ID does not exist"
Sep 16 17:29:19.919168 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.919127    2581 scope.go:117] "RemoveContainer" containerID="1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"
Sep 16 17:29:19.919347 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.919332    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"} err="failed to get container status \"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15\": rpc error: code = NotFound desc = could not find container \"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15\": container with ID starting with 1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15 not found: ID does not exist"
Sep 16 17:29:19.919347 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.919346    2581 scope.go:117] "RemoveContainer" containerID="87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"
Sep 16 17:29:19.919526 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.919511    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"} err="failed to get container status \"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed\": rpc error: code = NotFound desc = could not find container \"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed\": container with ID starting with 87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed not found: ID does not exist"
Sep 16 17:29:19.919561 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.919526    2581 scope.go:117] "RemoveContainer" containerID="902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"
Sep 16 17:29:19.919703 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.919689    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"} err="failed to get container status \"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0\": rpc error: code = NotFound desc = could not find container \"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0\": container with ID starting with 902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0 not found: ID does not exist"
Sep 16 17:29:19.919744 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.919703    2581 scope.go:117] "RemoveContainer" containerID="c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"
Sep 16 17:29:19.919955 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.919913    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"} err="failed to get container status \"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0\": rpc error: code = NotFound desc = could not find container \"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0\": container with ID starting with c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0 not found: ID does not exist"
Sep 16 17:29:19.919955 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.919929    2581 scope.go:117] "RemoveContainer" containerID="8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"
Sep 16 17:29:19.920099 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.920085    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"} err="failed to get container status \"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc\": rpc error: code = NotFound desc = could not find container \"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc\": container with ID starting with 8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc not found: ID does not exist"
Sep 16 17:29:19.920099 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.920098    2581 scope.go:117] "RemoveContainer" containerID="013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8"
Sep 16 17:29:19.920263 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.920248    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8"} err="failed to get container status \"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8\": rpc error: code = NotFound desc = could not find container \"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8\": container with ID starting with 013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8 not found: ID does not exist"
Sep 16 17:29:19.920302 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.920262    2581 scope.go:117] "RemoveContainer" containerID="2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"
Sep 16 17:29:19.920475 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.920460    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"} err="failed to get container status \"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401\": rpc error: code = NotFound desc = could not find container \"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401\": container with ID starting with 2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401 not found: ID does not exist"
Sep 16 17:29:19.920518 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.920475    2581 scope.go:117] "RemoveContainer" containerID="1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"
Sep 16 17:29:19.920681 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.920663    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"} err="failed to get container status \"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15\": rpc error: code = NotFound desc = could not find container \"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15\": container with ID starting with 1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15 not found: ID does not exist"
Sep 16 17:29:19.920681 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.920680    2581 scope.go:117] "RemoveContainer" containerID="87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"
Sep 16 17:29:19.920867 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.920853    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"} err="failed to get container status \"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed\": rpc error: code = NotFound desc = could not find container \"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed\": container with ID starting with 87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed not found: ID does not exist"
Sep 16 17:29:19.920927 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.920867    2581 scope.go:117] "RemoveContainer" containerID="902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"
Sep 16 17:29:19.921091 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.921075    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"} err="failed to get container status \"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0\": rpc error: code = NotFound desc = could not find container \"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0\": container with ID starting with 902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0 not found: ID does not exist"
Sep 16 17:29:19.921091 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.921090    2581 scope.go:117] "RemoveContainer" containerID="c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"
Sep 16 17:29:19.921258 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.921243    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"} err="failed to get container status \"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0\": rpc error: code = NotFound desc = could not find container \"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0\": container with ID starting with c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0 not found: ID does not exist"
Sep 16 17:29:19.921258 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.921257    2581 scope.go:117] "RemoveContainer" containerID="8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"
Sep 16 17:29:19.921468 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.921451    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"} err="failed to get container status \"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc\": rpc error: code = NotFound desc = could not find container \"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc\": container with ID starting with 8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc not found: ID does not exist"
Sep 16 17:29:19.921508 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.921469    2581 scope.go:117] "RemoveContainer" containerID="013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8"
Sep 16 17:29:19.921667 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.921652    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8"} err="failed to get container status \"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8\": rpc error: code = NotFound desc = could not find container \"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8\": container with ID starting with 013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8 not found: ID does not exist"
Sep 16 17:29:19.921702 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.921667    2581 scope.go:117] "RemoveContainer" containerID="2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"
Sep 16 17:29:19.921866 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.921852    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401"} err="failed to get container status \"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401\": rpc error: code = NotFound desc = could not find container \"2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401\": container with ID starting with 2c805f6d9db787c796f151022bf7f75d96d494920d690664fb2a681fd7186401 not found: ID does not exist"
Sep 16 17:29:19.921866 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.921865    2581 scope.go:117] "RemoveContainer" containerID="1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"
Sep 16 17:29:19.922077 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.922060    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15"} err="failed to get container status \"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15\": rpc error: code = NotFound desc = could not find container \"1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15\": container with ID starting with 1e5b182c1a452a5c1dcab1b422e0e6d1ece2f38da61f00e45a62f08877c75b15 not found: ID does not exist"
Sep 16 17:29:19.922115 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.922078    2581 scope.go:117] "RemoveContainer" containerID="87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"
Sep 16 17:29:19.922292 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.922277    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed"} err="failed to get container status \"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed\": rpc error: code = NotFound desc = could not find container \"87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed\": container with ID starting with 87c7559f45f335e07aa10d654d5ae689b0351241ec3d16a8a0503611c125a2ed not found: ID does not exist"
Sep 16 17:29:19.922342 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.922292    2581 scope.go:117] "RemoveContainer" containerID="902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"
Sep 16 17:29:19.922466 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.922450    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0"} err="failed to get container status \"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0\": rpc error: code = NotFound desc = could not find container \"902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0\": container with ID starting with 902dcda2c2f7501319b7726f000b4840ad284ad736da745a5d969fa6325d21b0 not found: ID does not exist"
Sep 16 17:29:19.922504 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.922466    2581 scope.go:117] "RemoveContainer" containerID="c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"
Sep 16 17:29:19.922682 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.922665    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0"} err="failed to get container status \"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0\": rpc error: code = NotFound desc = could not find container \"c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0\": container with ID starting with c7682d3325048f0bb6ae559c0929776af1407654c35e308dde9546bfdfdf9de0 not found: ID does not exist"
Sep 16 17:29:19.922733 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.922681    2581 scope.go:117] "RemoveContainer" containerID="8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"
Sep 16 17:29:19.922870 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.922856    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc"} err="failed to get container status \"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc\": rpc error: code = NotFound desc = could not find container \"8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc\": container with ID starting with 8ba4923b304f526ef99af5056ac5df974b8685b66de5b208d7de47421e083ddc not found: ID does not exist"
Sep 16 17:29:19.922960 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.922870    2581 scope.go:117] "RemoveContainer" containerID="013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8"
Sep 16 17:29:19.923122 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.923108    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8"} err="failed to get container status \"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8\": rpc error: code = NotFound desc = could not find container \"013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8\": container with ID starting with 013101cc8a85f80c03efe424014d800c17dccfd7a6b40599f2dc414c03ad85e8 not found: ID does not exist"
Sep 16 17:29:19.935638 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.935614    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:29:19.935921 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.935909    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="thanos-sidecar"
Sep 16 17:29:19.935973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.935924    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="thanos-sidecar"
Sep 16 17:29:19.935973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.935933    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="kube-rbac-proxy-thanos"
Sep 16 17:29:19.935973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.935939    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="kube-rbac-proxy-thanos"
Sep 16 17:29:19.935973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.935946    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="config-reloader"
Sep 16 17:29:19.935973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.935951    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="config-reloader"
Sep 16 17:29:19.935973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.935962    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="init-config-reloader"
Sep 16 17:29:19.935973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.935967    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="init-config-reloader"
Sep 16 17:29:19.935973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.935972    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="kube-rbac-proxy"
Sep 16 17:29:19.936185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.935977    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="kube-rbac-proxy"
Sep 16 17:29:19.936185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.935984    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="prometheus"
Sep 16 17:29:19.936185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.935989    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="prometheus"
Sep 16 17:29:19.936185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.935994    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="kube-rbac-proxy-web"
Sep 16 17:29:19.936185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.935998    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="kube-rbac-proxy-web"
Sep 16 17:29:19.936185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.936048    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="prometheus"
Sep 16 17:29:19.936185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.936055    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="config-reloader"
Sep 16 17:29:19.936185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.936061    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="thanos-sidecar"
Sep 16 17:29:19.936185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.936067    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="kube-rbac-proxy"
Sep 16 17:29:19.936185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.936072    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="kube-rbac-proxy-web"
Sep 16 17:29:19.936185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.936076    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7983b215-4d22-4c1c-9254-24464938f77e" containerName="kube-rbac-proxy-thanos"
Sep 16 17:29:19.940972 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.940956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.944166 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.944144    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 17:29:19.944265 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.944214    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 17:29:19.944358 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.944339    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 17:29:19.944653 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.944633    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 17:29:19.944718 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.944645    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 17:29:19.944718 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.944641    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 17:29:19.944821 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.944786    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-537248iqg2jv2\""
Sep 16 17:29:19.945040 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.945021    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 17:29:19.945102 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.945039    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 17:29:19.945429 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.945409    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-b9p8j\""
Sep 16 17:29:19.945536 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.945511    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 17:29:19.945658 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.945643    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 17:29:19.945908 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.945879    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 17:29:19.945982 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.945959    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 17:29:19.946652 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.946633    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 17:29:19.947804 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.947785    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 17:29:19.950221 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.950204    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 17:29:19.959527 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.957209    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:29:19.999035 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999035 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999221 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-web-config\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999221 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999108    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cca209e4-a21d-4f71-9146-c49b09985738-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999221 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999125    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999221 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999221 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999188    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999385 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999385 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csqgt\" (UniqueName: \"kubernetes.io/projected/cca209e4-a21d-4f71-9146-c49b09985738-kube-api-access-csqgt\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999385 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999385 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999385 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-config\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999385 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999547 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999547 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cca209e4-a21d-4f71-9146-c49b09985738-config-out\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999547 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999674 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cca209e4-a21d-4f71-9146-c49b09985738-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999674 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999766 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999672    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:19.999766 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:19.999700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.100511 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.100511 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csqgt\" (UniqueName: \"kubernetes.io/projected/cca209e4-a21d-4f71-9146-c49b09985738-kube-api-access-csqgt\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.100511 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.100511 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.100819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-config\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.100819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.100819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.100819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cca209e4-a21d-4f71-9146-c49b09985738-config-out\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.100819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.100819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cca209e4-a21d-4f71-9146-c49b09985738-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.100819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.100819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.100819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.100819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.100819 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.101347 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-web-config\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.101347 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cca209e4-a21d-4f71-9146-c49b09985738-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.101347 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.101347 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.100968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.101347 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.101003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.101347 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.101135    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cca209e4-a21d-4f71-9146-c49b09985738-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.101646 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.101471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.103465 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.101792    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.103465 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.102449    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.103465 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.103361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.104194 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.103819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-config\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.104194 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.103984    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.104194 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.104005    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.104194 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.104094    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.104665 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.104255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.104665 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.104281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.104665 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.104379    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cca209e4-a21d-4f71-9146-c49b09985738-config-out\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.105080 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.104719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.105080 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.104802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.106072 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.106048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.106402 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.106387    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cca209e4-a21d-4f71-9146-c49b09985738-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.107073 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.107053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-web-config\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.107113 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.107101    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cca209e4-a21d-4f71-9146-c49b09985738-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.107531 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.107512    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cca209e4-a21d-4f71-9146-c49b09985738-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.109263 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.109242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csqgt\" (UniqueName: \"kubernetes.io/projected/cca209e4-a21d-4f71-9146-c49b09985738-kube-api-access-csqgt\") pod \"prometheus-k8s-0\" (UID: \"cca209e4-a21d-4f71-9146-c49b09985738\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.251826 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.251788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:20.533973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.533945    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:29:20.534806 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:29:20.534781    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca209e4_a21d_4f71_9146_c49b09985738.slice/crio-eedcffb4f8147ac7bbaa4fcce3e05f4b5399469ef8397818f70ba53f5a02550c WatchSource:0}: Error finding container eedcffb4f8147ac7bbaa4fcce3e05f4b5399469ef8397818f70ba53f5a02550c: Status 404 returned error can't find the container with id eedcffb4f8147ac7bbaa4fcce3e05f4b5399469ef8397818f70ba53f5a02550c
Sep 16 17:29:20.872469 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.872391    2581 generic.go:356] "Generic (PLEG): container finished" podID="cca209e4-a21d-4f71-9146-c49b09985738" containerID="6c8ab1111b9eb028b2f634748f7ead7b949a86f10abb92bf84db3c3429576a4c" exitCode=0
Sep 16 17:29:20.872629 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.872468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cca209e4-a21d-4f71-9146-c49b09985738","Type":"ContainerDied","Data":"6c8ab1111b9eb028b2f634748f7ead7b949a86f10abb92bf84db3c3429576a4c"}
Sep 16 17:29:20.872629 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:20.872530    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cca209e4-a21d-4f71-9146-c49b09985738","Type":"ContainerStarted","Data":"eedcffb4f8147ac7bbaa4fcce3e05f4b5399469ef8397818f70ba53f5a02550c"}
Sep 16 17:29:21.501130 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:21.501046    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:21.602346 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:21.602262    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:21.768633 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:21.768543    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7983b215-4d22-4c1c-9254-24464938f77e" path="/var/lib/kubelet/pods/7983b215-4d22-4c1c-9254-24464938f77e/volumes"
Sep 16 17:29:21.882709 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:21.882673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cca209e4-a21d-4f71-9146-c49b09985738","Type":"ContainerStarted","Data":"41f29e564543a53a4595676357d930ee02c2fcbf25a4ed4a4c7edaa96d1e1bc4"}
Sep 16 17:29:21.882867 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:21.882747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cca209e4-a21d-4f71-9146-c49b09985738","Type":"ContainerStarted","Data":"e884f6d1787a549addfd30b73a8036f5f9abc86047b74312840b22f06515260e"}
Sep 16 17:29:21.882867 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:21.882784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cca209e4-a21d-4f71-9146-c49b09985738","Type":"ContainerStarted","Data":"304d6e5023b9e00af852836896725c117d3fd9b2f3a020bc68680420d09bdd08"}
Sep 16 17:29:21.882867 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:21.882821    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cca209e4-a21d-4f71-9146-c49b09985738","Type":"ContainerStarted","Data":"931029c535362e2c4cce45f4f3fccc241b7761b8e1dddf85cec485dd9f6619b2"}
Sep 16 17:29:21.882867 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:21.882858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cca209e4-a21d-4f71-9146-c49b09985738","Type":"ContainerStarted","Data":"1c04e5b4006f4c08fe5e5e4a53937610b8abe6f575fd46343465cce27e804cd9"}
Sep 16 17:29:21.883110 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:21.882916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cca209e4-a21d-4f71-9146-c49b09985738","Type":"ContainerStarted","Data":"f9dd90da6c64f7fd8a7fbbb71157b714af7b00f0affad82955f17e3097261026"}
Sep 16 17:29:21.919375 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:21.919303    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.89724212 podStartE2EDuration="2.919283858s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.041738ms" isStatefulPod=false podCreationTimestamp="2026-09-16 17:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:29:21.918066239 +0000 UTC m=+290.706688774" watchObservedRunningTime="2026-09-16 17:29:21.919283858 +0000 UTC m=+290.707906389"
Sep 16 17:29:25.252290 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:25.252255    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:29.876151 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:29.876124    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-klh2p"
Sep 16 17:29:31.650709 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:31.650681    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:29:31.657047 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:31.657024    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:29:31.661937 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:31.661918    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 17:29:32.786255 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:32.786206    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:32.887064 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:32.887020    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:33.094836 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:33.094735    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:39.005616 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:39.005584    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:39.079537 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:29:39.079501    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:07.255555 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:07.255515    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:07.356828 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:07.356789    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:20.252709 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:20.252670    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:30:20.268692 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:20.268666    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:30:21.136091 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:21.136063    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:30:28.694810 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:28.694715    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:28.745075 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:28.745041    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:39.040294 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:39.040254    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:39.134329 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:39.134275    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:39.235384 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:39.235343    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:39.336115 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:39.336022    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:39.741775 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:39.741664    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:39.796805 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:39.796778    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:40.674701 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:40.674667    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:40.727229 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:40.727182    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:44.409354 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:44.409304    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:44.510872 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:44.510837    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:44.561592 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:44.561548    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:50.242713 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:50.242669    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:50.321524 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:30:50.321484    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:31:29.452695 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:31:29.452657    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:31:42.088613 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:31:42.088574    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:31:42.143952 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:31:42.143921    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:31:55.335379 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:31:55.335287    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:31:55.447879 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:31:55.447839    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:31:55.503679 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:31:55.503651    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:17.811352 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:32:17.811311    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:18.877837 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:32:18.877808    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:24.898697 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:32:24.898656    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:25.213484 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:32:25.213391    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:25.361291 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:32:25.361254    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:25.454171 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:32:25.454139    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:25.516425 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:32:25.516329    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:26.083508 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:32:26.083471    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:26.178664 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:32:26.178623    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:26.245375 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:32:26.245335    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:26.459662 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:32:26.459568    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:26.637969 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:32:26.637917    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:26.816323 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:32:26.816281    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:26.913870 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:32:26.913833    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:17.093792 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:33:17.093755    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:17.191843 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:33:17.191800    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:21.554870 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:33:21.554829    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:32.967231 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:33:32.967149    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:33.057080 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:33:33.057046    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:48.666769 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:33:48.666735    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:50.839063 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:33:50.839031    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:50.937684 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:33:50.937646    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:51.046394 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:33:51.046356    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:51.129449 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:33:51.129362    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:51.219569 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:33:51.219533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:51.325157 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:33:51.325112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:51.668774 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:33:51.668740    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:51.766794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:33:51.766762    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:51.823198 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:33:51.823158    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:15.900600 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:34:15.900561    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:16.001288 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:34:16.001248    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:31.678134 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:34:31.678105    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:34:31.682679 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:34:31.682656    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:34:31.711825 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:34:31.711803    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:31.810992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:34:31.810965    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:13.352467 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:13.352389    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:13.456521 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:13.456491    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:13.634629 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:13.634564    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:13.733592 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:13.733563    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:13.912227 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:13.912161    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:14.829747 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:14.829716    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:14.931048 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:14.931014    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:29.879058 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:29.879026    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-bjxl7"]
Sep 16 17:35:29.882217 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:29.882201    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bjxl7"
Sep 16 17:35:29.884908 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:29.884874    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 17:35:29.885012 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:29.884886    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 17:35:29.885012 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:29.884946    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 17:35:29.885128 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:29.885030    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 17:35:29.886028 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:29.886008    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hlr6s\""
Sep 16 17:35:29.890644 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:29.890626    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bjxl7"]
Sep 16 17:35:29.960274 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:29.960254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f5129f-6c92-41bf-92e4-79da60036a74-model-cache\") pod \"s3-init-bjxl7\" (UID: \"a7f5129f-6c92-41bf-92e4-79da60036a74\") " pod="kserve/s3-init-bjxl7"
Sep 16 17:35:29.960357 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:29.960280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtrcz\" (UniqueName: \"kubernetes.io/projected/a7f5129f-6c92-41bf-92e4-79da60036a74-kube-api-access-vtrcz\") pod \"s3-init-bjxl7\" (UID: \"a7f5129f-6c92-41bf-92e4-79da60036a74\") " pod="kserve/s3-init-bjxl7"
Sep 16 17:35:30.061145 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:30.061125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f5129f-6c92-41bf-92e4-79da60036a74-model-cache\") pod \"s3-init-bjxl7\" (UID: \"a7f5129f-6c92-41bf-92e4-79da60036a74\") " pod="kserve/s3-init-bjxl7"
Sep 16 17:35:30.061236 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:30.061152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtrcz\" (UniqueName: \"kubernetes.io/projected/a7f5129f-6c92-41bf-92e4-79da60036a74-kube-api-access-vtrcz\") pod \"s3-init-bjxl7\" (UID: \"a7f5129f-6c92-41bf-92e4-79da60036a74\") " pod="kserve/s3-init-bjxl7"
Sep 16 17:35:30.061421 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:30.061403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f5129f-6c92-41bf-92e4-79da60036a74-model-cache\") pod \"s3-init-bjxl7\" (UID: \"a7f5129f-6c92-41bf-92e4-79da60036a74\") " pod="kserve/s3-init-bjxl7"
Sep 16 17:35:30.069765 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:30.069738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtrcz\" (UniqueName: \"kubernetes.io/projected/a7f5129f-6c92-41bf-92e4-79da60036a74-kube-api-access-vtrcz\") pod \"s3-init-bjxl7\" (UID: \"a7f5129f-6c92-41bf-92e4-79da60036a74\") " pod="kserve/s3-init-bjxl7"
Sep 16 17:35:30.203235 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:30.203185    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bjxl7"
Sep 16 17:35:30.478693 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:30.478634    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bjxl7"]
Sep 16 17:35:30.480658 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:35:30.480633    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7f5129f_6c92_41bf_92e4_79da60036a74.slice/crio-833af86fad37635cb081ae42b1fa3c069d549445a4120f1d36a84a55e61f4e75 WatchSource:0}: Error finding container 833af86fad37635cb081ae42b1fa3c069d549445a4120f1d36a84a55e61f4e75: Status 404 returned error can't find the container with id 833af86fad37635cb081ae42b1fa3c069d549445a4120f1d36a84a55e61f4e75
Sep 16 17:35:30.482453 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:30.482436    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:35:31.318790 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:31.318757    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bjxl7" event={"ID":"a7f5129f-6c92-41bf-92e4-79da60036a74","Type":"ContainerStarted","Data":"833af86fad37635cb081ae42b1fa3c069d549445a4120f1d36a84a55e61f4e75"}
Sep 16 17:35:34.412015 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:34.411994    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 17:35:35.338709 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:35.338660    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bjxl7" event={"ID":"a7f5129f-6c92-41bf-92e4-79da60036a74","Type":"ContainerStarted","Data":"ff4ce2419d16c4d9701e6f5b73f3942d7dbf36bb6115081685b520c1d04f1049"}
Sep 16 17:35:36.344230 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:36.344197    2581 generic.go:356] "Generic (PLEG): container finished" podID="a7f5129f-6c92-41bf-92e4-79da60036a74" containerID="ff4ce2419d16c4d9701e6f5b73f3942d7dbf36bb6115081685b520c1d04f1049" exitCode=0
Sep 16 17:35:36.344610 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:36.344238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bjxl7" event={"ID":"a7f5129f-6c92-41bf-92e4-79da60036a74","Type":"ContainerDied","Data":"ff4ce2419d16c4d9701e6f5b73f3942d7dbf36bb6115081685b520c1d04f1049"}
Sep 16 17:35:40.682490 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:40.682468    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 17:35:41.371820 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:41.371784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bjxl7" event={"ID":"a7f5129f-6c92-41bf-92e4-79da60036a74","Type":"ContainerStarted","Data":"b90faed967c9ae327eaf3f594de0f172ab93008f9f37f99dee97c0ee2fd69cb5"}
Sep 16 17:35:41.404867 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:35:41.404815    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-bjxl7" podStartSLOduration=2.705638924 podStartE2EDuration="12.40480185s" totalImagesPullingTime="8.250536782s" totalInitContainerRuntime="1.448626144s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:35:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:35:41.404450936 +0000 UTC m=+670.193073467" watchObservedRunningTime="2026-09-16 17:35:41.40480185 +0000 UTC m=+670.193424381"
Sep 16 17:36:45.605647 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:36:45.605587    2581 generic.go:356] "Generic (PLEG): container finished" podID="a7f5129f-6c92-41bf-92e4-79da60036a74" containerID="b90faed967c9ae327eaf3f594de0f172ab93008f9f37f99dee97c0ee2fd69cb5" exitCode=0
Sep 16 17:36:45.605996 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:36:45.605652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bjxl7" event={"ID":"a7f5129f-6c92-41bf-92e4-79da60036a74","Type":"ContainerDied","Data":"b90faed967c9ae327eaf3f594de0f172ab93008f9f37f99dee97c0ee2fd69cb5"}
Sep 16 17:36:46.720117 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:36:46.720093    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bjxl7"
Sep 16 17:36:46.795309 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:36:46.795285    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtrcz\" (UniqueName: \"kubernetes.io/projected/a7f5129f-6c92-41bf-92e4-79da60036a74-kube-api-access-vtrcz\") pod \"a7f5129f-6c92-41bf-92e4-79da60036a74\" (UID: \"a7f5129f-6c92-41bf-92e4-79da60036a74\") "
Sep 16 17:36:46.795406 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:36:46.795367    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f5129f-6c92-41bf-92e4-79da60036a74-model-cache\") pod \"a7f5129f-6c92-41bf-92e4-79da60036a74\" (UID: \"a7f5129f-6c92-41bf-92e4-79da60036a74\") "
Sep 16 17:36:46.797317 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:36:46.797293    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f5129f-6c92-41bf-92e4-79da60036a74-kube-api-access-vtrcz" (OuterVolumeSpecName: "kube-api-access-vtrcz") pod "a7f5129f-6c92-41bf-92e4-79da60036a74" (UID: "a7f5129f-6c92-41bf-92e4-79da60036a74"). InnerVolumeSpecName "kube-api-access-vtrcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:36:46.812743 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:36:46.812714    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f5129f-6c92-41bf-92e4-79da60036a74-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7f5129f-6c92-41bf-92e4-79da60036a74" (UID: "a7f5129f-6c92-41bf-92e4-79da60036a74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:36:46.896095 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:36:46.896046    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f5129f-6c92-41bf-92e4-79da60036a74-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:36:46.896095 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:36:46.896065    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtrcz\" (UniqueName: \"kubernetes.io/projected/a7f5129f-6c92-41bf-92e4-79da60036a74-kube-api-access-vtrcz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:36:47.614333 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:36:47.614308    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bjxl7"
Sep 16 17:36:47.614562 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:36:47.614336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bjxl7" event={"ID":"a7f5129f-6c92-41bf-92e4-79da60036a74","Type":"ContainerDied","Data":"833af86fad37635cb081ae42b1fa3c069d549445a4120f1d36a84a55e61f4e75"}
Sep 16 17:36:47.614562 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:36:47.614367    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="833af86fad37635cb081ae42b1fa3c069d549445a4120f1d36a84a55e61f4e75"
Sep 16 17:36:48.377818 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:36:48.377787    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:49.164596 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:36:49.164559    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:21.910686 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:21.910651    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-l4fbb"]
Sep 16 17:37:21.911250 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:21.911080    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f5129f-6c92-41bf-92e4-79da60036a74" containerName="s3-init"
Sep 16 17:37:21.911250 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:21.911099    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f5129f-6c92-41bf-92e4-79da60036a74" containerName="s3-init"
Sep 16 17:37:21.911250 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:21.911114    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f5129f-6c92-41bf-92e4-79da60036a74" containerName="download-hf-model"
Sep 16 17:37:21.911250 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:21.911122    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f5129f-6c92-41bf-92e4-79da60036a74" containerName="download-hf-model"
Sep 16 17:37:21.911250 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:21.911220    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f5129f-6c92-41bf-92e4-79da60036a74" containerName="s3-init"
Sep 16 17:37:21.914207 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:21.914187    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-l4fbb"
Sep 16 17:37:21.917064 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:21.917041    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 17:37:21.917187 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:21.917069    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 17:37:21.917187 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:21.917102    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 17:37:21.917187 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:21.917129    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 16 17:37:21.917333 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:21.917190    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hlr6s\""
Sep 16 17:37:21.923386 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:21.923355    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-l4fbb"]
Sep 16 17:37:22.034783 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:22.034760    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkrvm\" (UniqueName: \"kubernetes.io/projected/e7724395-44a9-45f6-bbc3-ac2513b0db86-kube-api-access-xkrvm\") pod \"s3-tls-init-custom-l4fbb\" (UID: \"e7724395-44a9-45f6-bbc3-ac2513b0db86\") " pod="kserve/s3-tls-init-custom-l4fbb"
Sep 16 17:37:22.034883 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:22.034795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7724395-44a9-45f6-bbc3-ac2513b0db86-model-cache\") pod \"s3-tls-init-custom-l4fbb\" (UID: \"e7724395-44a9-45f6-bbc3-ac2513b0db86\") " pod="kserve/s3-tls-init-custom-l4fbb"
Sep 16 17:37:22.135338 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:22.135317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkrvm\" (UniqueName: \"kubernetes.io/projected/e7724395-44a9-45f6-bbc3-ac2513b0db86-kube-api-access-xkrvm\") pod \"s3-tls-init-custom-l4fbb\" (UID: \"e7724395-44a9-45f6-bbc3-ac2513b0db86\") " pod="kserve/s3-tls-init-custom-l4fbb"
Sep 16 17:37:22.135420 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:22.135350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7724395-44a9-45f6-bbc3-ac2513b0db86-model-cache\") pod \"s3-tls-init-custom-l4fbb\" (UID: \"e7724395-44a9-45f6-bbc3-ac2513b0db86\") " pod="kserve/s3-tls-init-custom-l4fbb"
Sep 16 17:37:22.135684 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:22.135668    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7724395-44a9-45f6-bbc3-ac2513b0db86-model-cache\") pod \"s3-tls-init-custom-l4fbb\" (UID: \"e7724395-44a9-45f6-bbc3-ac2513b0db86\") " pod="kserve/s3-tls-init-custom-l4fbb"
Sep 16 17:37:22.144654 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:22.144636    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkrvm\" (UniqueName: \"kubernetes.io/projected/e7724395-44a9-45f6-bbc3-ac2513b0db86-kube-api-access-xkrvm\") pod \"s3-tls-init-custom-l4fbb\" (UID: \"e7724395-44a9-45f6-bbc3-ac2513b0db86\") " pod="kserve/s3-tls-init-custom-l4fbb"
Sep 16 17:37:22.231305 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:22.231260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-l4fbb"
Sep 16 17:37:22.499780 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:22.499701    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-l4fbb"]
Sep 16 17:37:22.501981 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:37:22.501956    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7724395_44a9_45f6_bbc3_ac2513b0db86.slice/crio-42ca856457609169f0cf608af8c7dcc573c224f6fa7c95940cb81efa2d825184 WatchSource:0}: Error finding container 42ca856457609169f0cf608af8c7dcc573c224f6fa7c95940cb81efa2d825184: Status 404 returned error can't find the container with id 42ca856457609169f0cf608af8c7dcc573c224f6fa7c95940cb81efa2d825184
Sep 16 17:37:22.736976 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:22.736937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-l4fbb" event={"ID":"e7724395-44a9-45f6-bbc3-ac2513b0db86","Type":"ContainerStarted","Data":"817a64490b798b2bc755f0ef697250a12966a3213a550b8f87b82087edc20a92"}
Sep 16 17:37:22.737141 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:22.736984    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-l4fbb" event={"ID":"e7724395-44a9-45f6-bbc3-ac2513b0db86","Type":"ContainerStarted","Data":"42ca856457609169f0cf608af8c7dcc573c224f6fa7c95940cb81efa2d825184"}
Sep 16 17:37:24.746418 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:24.746387    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7724395-44a9-45f6-bbc3-ac2513b0db86" containerID="817a64490b798b2bc755f0ef697250a12966a3213a550b8f87b82087edc20a92" exitCode=0
Sep 16 17:37:24.746700 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:24.746458    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-l4fbb" event={"ID":"e7724395-44a9-45f6-bbc3-ac2513b0db86","Type":"ContainerDied","Data":"817a64490b798b2bc755f0ef697250a12966a3213a550b8f87b82087edc20a92"}
Sep 16 17:37:25.751689 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:25.751658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-l4fbb" event={"ID":"e7724395-44a9-45f6-bbc3-ac2513b0db86","Type":"ContainerStarted","Data":"f91d1c64ab04b06dbd181e8fcb29ae7ed92bba1c3b0fea65edf57d5ff0c08540"}
Sep 16 17:37:25.771523 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:25.771481    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-l4fbb" podStartSLOduration=3.141040854 podStartE2EDuration="4.771467402s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.630426548s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:37:25.770813323 +0000 UTC m=+774.559435867" watchObservedRunningTime="2026-09-16 17:37:25.771467402 +0000 UTC m=+774.560089933"
Sep 16 17:37:31.778401 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:31.778373    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7724395-44a9-45f6-bbc3-ac2513b0db86" containerID="f91d1c64ab04b06dbd181e8fcb29ae7ed92bba1c3b0fea65edf57d5ff0c08540" exitCode=0
Sep 16 17:37:31.778713 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:31.778415    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-l4fbb" event={"ID":"e7724395-44a9-45f6-bbc3-ac2513b0db86","Type":"ContainerDied","Data":"f91d1c64ab04b06dbd181e8fcb29ae7ed92bba1c3b0fea65edf57d5ff0c08540"}
Sep 16 17:37:32.909402 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:32.909378    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-l4fbb"
Sep 16 17:37:32.923454 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:32.923430    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkrvm\" (UniqueName: \"kubernetes.io/projected/e7724395-44a9-45f6-bbc3-ac2513b0db86-kube-api-access-xkrvm\") pod \"e7724395-44a9-45f6-bbc3-ac2513b0db86\" (UID: \"e7724395-44a9-45f6-bbc3-ac2513b0db86\") "
Sep 16 17:37:32.923516 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:32.923477    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7724395-44a9-45f6-bbc3-ac2513b0db86-model-cache\") pod \"e7724395-44a9-45f6-bbc3-ac2513b0db86\" (UID: \"e7724395-44a9-45f6-bbc3-ac2513b0db86\") "
Sep 16 17:37:32.925438 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:32.925409    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7724395-44a9-45f6-bbc3-ac2513b0db86-kube-api-access-xkrvm" (OuterVolumeSpecName: "kube-api-access-xkrvm") pod "e7724395-44a9-45f6-bbc3-ac2513b0db86" (UID: "e7724395-44a9-45f6-bbc3-ac2513b0db86"). InnerVolumeSpecName "kube-api-access-xkrvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:37:32.946485 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:32.946452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7724395-44a9-45f6-bbc3-ac2513b0db86-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7724395-44a9-45f6-bbc3-ac2513b0db86" (UID: "e7724395-44a9-45f6-bbc3-ac2513b0db86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:37:33.024480 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:33.024454    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkrvm\" (UniqueName: \"kubernetes.io/projected/e7724395-44a9-45f6-bbc3-ac2513b0db86-kube-api-access-xkrvm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:37:33.024480 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:33.024477    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7724395-44a9-45f6-bbc3-ac2513b0db86-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:37:33.786995 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:33.786974    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-l4fbb" event={"ID":"e7724395-44a9-45f6-bbc3-ac2513b0db86","Type":"ContainerDied","Data":"42ca856457609169f0cf608af8c7dcc573c224f6fa7c95940cb81efa2d825184"}
Sep 16 17:37:33.786995 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:33.786988    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-l4fbb"
Sep 16 17:37:33.787128 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:33.787009    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42ca856457609169f0cf608af8c7dcc573c224f6fa7c95940cb81efa2d825184"
Sep 16 17:37:34.440790 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:34.440755    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:35.870454 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:35.870420    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:37.139920 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.139875    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-x5j5n"]
Sep 16 17:37:37.140243 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.140190    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7724395-44a9-45f6-bbc3-ac2513b0db86" containerName="s3-tls-init-custom"
Sep 16 17:37:37.140243 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.140202    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7724395-44a9-45f6-bbc3-ac2513b0db86" containerName="s3-tls-init-custom"
Sep 16 17:37:37.140243 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.140210    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7724395-44a9-45f6-bbc3-ac2513b0db86" containerName="download-hf-model"
Sep 16 17:37:37.140243 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.140215    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7724395-44a9-45f6-bbc3-ac2513b0db86" containerName="download-hf-model"
Sep 16 17:37:37.140366 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.140264    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7724395-44a9-45f6-bbc3-ac2513b0db86" containerName="s3-tls-init-custom"
Sep 16 17:37:37.144121 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.144106    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-x5j5n"
Sep 16 17:37:37.146843 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.146822    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 16 17:37:37.146966 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.146879    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 17:37:37.146966 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.146883    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 17:37:37.147172 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.147157    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 17:37:37.148424 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.148405    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hlr6s\""
Sep 16 17:37:37.150995 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.150975    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-x5j5n"]
Sep 16 17:37:37.151091 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.151017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxfft\" (UniqueName: \"kubernetes.io/projected/cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40-kube-api-access-kxfft\") pod \"s3-tls-init-serving-x5j5n\" (UID: \"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40\") " pod="kserve/s3-tls-init-serving-x5j5n"
Sep 16 17:37:37.151091 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.151054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40-model-cache\") pod \"s3-tls-init-serving-x5j5n\" (UID: \"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40\") " pod="kserve/s3-tls-init-serving-x5j5n"
Sep 16 17:37:37.251319 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.251298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxfft\" (UniqueName: \"kubernetes.io/projected/cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40-kube-api-access-kxfft\") pod \"s3-tls-init-serving-x5j5n\" (UID: \"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40\") " pod="kserve/s3-tls-init-serving-x5j5n"
Sep 16 17:37:37.251419 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.251326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40-model-cache\") pod \"s3-tls-init-serving-x5j5n\" (UID: \"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40\") " pod="kserve/s3-tls-init-serving-x5j5n"
Sep 16 17:37:37.251605 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.251590    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40-model-cache\") pod \"s3-tls-init-serving-x5j5n\" (UID: \"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40\") " pod="kserve/s3-tls-init-serving-x5j5n"
Sep 16 17:37:37.261006 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.260987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxfft\" (UniqueName: \"kubernetes.io/projected/cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40-kube-api-access-kxfft\") pod \"s3-tls-init-serving-x5j5n\" (UID: \"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40\") " pod="kserve/s3-tls-init-serving-x5j5n"
Sep 16 17:37:37.468507 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.468448    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-x5j5n"
Sep 16 17:37:37.738971 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.738905    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-x5j5n"]
Sep 16 17:37:37.741114 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:37:37.741081    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfcdbd4e_b7fe_44e3_96fe_17f2e9443e40.slice/crio-52cfe65d4057bd2b1e5b05c8da6c327c1a5d719cd9bd003103c58f5e0989540b WatchSource:0}: Error finding container 52cfe65d4057bd2b1e5b05c8da6c327c1a5d719cd9bd003103c58f5e0989540b: Status 404 returned error can't find the container with id 52cfe65d4057bd2b1e5b05c8da6c327c1a5d719cd9bd003103c58f5e0989540b
Sep 16 17:37:37.807822 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.807794    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-x5j5n" event={"ID":"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40","Type":"ContainerStarted","Data":"a029e62fefdc9a3613778d1f0870254adba6abec9040e2d4fe0db5b7ad12dc4b"}
Sep 16 17:37:37.807929 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:37.807839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-x5j5n" event={"ID":"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40","Type":"ContainerStarted","Data":"52cfe65d4057bd2b1e5b05c8da6c327c1a5d719cd9bd003103c58f5e0989540b"}
Sep 16 17:37:39.818090 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:39.818062    2581 generic.go:356] "Generic (PLEG): container finished" podID="cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40" containerID="a029e62fefdc9a3613778d1f0870254adba6abec9040e2d4fe0db5b7ad12dc4b" exitCode=0
Sep 16 17:37:39.818418 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:39.818129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-x5j5n" event={"ID":"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40","Type":"ContainerDied","Data":"a029e62fefdc9a3613778d1f0870254adba6abec9040e2d4fe0db5b7ad12dc4b"}
Sep 16 17:37:40.246518 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:40.246450    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:40.824019 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:40.823989    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-x5j5n" event={"ID":"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40","Type":"ContainerStarted","Data":"6de0726434957ec0ee50d369c651f243e58699b6c138e75d0be596ce27355389"}
Sep 16 17:37:40.849577 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:40.849535    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-x5j5n" podStartSLOduration=2.526293609 podStartE2EDuration="3.849521157s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.323227548s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:37:40.847286558 +0000 UTC m=+789.635909091" watchObservedRunningTime="2026-09-16 17:37:40.849521157 +0000 UTC m=+789.638143688"
Sep 16 17:37:46.848861 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:46.848831    2581 generic.go:356] "Generic (PLEG): container finished" podID="cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40" containerID="6de0726434957ec0ee50d369c651f243e58699b6c138e75d0be596ce27355389" exitCode=0
Sep 16 17:37:46.849214 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:46.848920    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-x5j5n" event={"ID":"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40","Type":"ContainerDied","Data":"6de0726434957ec0ee50d369c651f243e58699b6c138e75d0be596ce27355389"}
Sep 16 17:37:47.979196 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:47.979175    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-x5j5n"
Sep 16 17:37:48.041318 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:48.041294    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxfft\" (UniqueName: \"kubernetes.io/projected/cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40-kube-api-access-kxfft\") pod \"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40\" (UID: \"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40\") "
Sep 16 17:37:48.041425 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:48.041345    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40-model-cache\") pod \"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40\" (UID: \"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40\") "
Sep 16 17:37:48.043288 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:48.043259    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40-kube-api-access-kxfft" (OuterVolumeSpecName: "kube-api-access-kxfft") pod "cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40" (UID: "cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40"). InnerVolumeSpecName "kube-api-access-kxfft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:37:48.063118 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:48.063083    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40" (UID: "cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:37:48.141984 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:48.141929    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:37:48.141984 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:48.141953    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxfft\" (UniqueName: \"kubernetes.io/projected/cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40-kube-api-access-kxfft\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:37:48.858613 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:48.858597    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-x5j5n"
Sep 16 17:37:48.858709 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:48.858627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-x5j5n" event={"ID":"cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40","Type":"ContainerDied","Data":"52cfe65d4057bd2b1e5b05c8da6c327c1a5d719cd9bd003103c58f5e0989540b"}
Sep 16 17:37:48.858709 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:48.858661    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52cfe65d4057bd2b1e5b05c8da6c327c1a5d719cd9bd003103c58f5e0989540b"
Sep 16 17:37:49.350160 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:49.350132    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:49.453774 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:37:49.453748    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:01.814923 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.814835    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:01.933133 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.933109    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"]
Sep 16 17:38:01.933443 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.933430    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40" containerName="s3-tls-init-serving"
Sep 16 17:38:01.933493 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.933445    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40" containerName="s3-tls-init-serving"
Sep 16 17:38:01.933493 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.933456    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40" containerName="download-hf-model"
Sep 16 17:38:01.933493 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.933461    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40" containerName="download-hf-model"
Sep 16 17:38:01.933588 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.933517    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40" containerName="s3-tls-init-serving"
Sep 16 17:38:01.936594 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.936579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:01.939262 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.939244    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 17:38:01.939477 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.939456    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 17:38:01.939549 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.939505    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 17:38:01.939607 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.939584    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:38:01.939712 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.939695    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\""
Sep 16 17:38:01.939776 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.939730    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 16 17:38:01.940394 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.940377    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 16 17:38:01.950731 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:01.950708    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"]
Sep 16 17:38:02.023460 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.023425    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:02.033566 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.033546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-4h7qs\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.033655 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.033582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-4h7qs\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.033655 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.033620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrg7j\" (UniqueName: \"kubernetes.io/projected/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kube-api-access-lrg7j\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-4h7qs\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.033722 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.033687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-4h7qs\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.033751 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.033720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-4h7qs\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.061092 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.061072    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"]
Sep 16 17:38:02.064305 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.064289    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.067381 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.067335    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-a427f5-predictor-serving-cert\""
Sep 16 17:38:02.067637 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.067619    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-a427f5-dockercfg-fm74p\""
Sep 16 17:38:02.067715 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.067621    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-a427f5\""
Sep 16 17:38:02.067715 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.067698    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-a427f5-kube-rbac-proxy-sar-config\""
Sep 16 17:38:02.080999 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.080982    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"]
Sep 16 17:38:02.103788 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.103758    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:02.135090 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.135062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btqnd\" (UniqueName: \"kubernetes.io/projected/e977f591-eb58-419c-9afe-5f391cf1d8e4-kube-api-access-btqnd\") pod \"isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") " pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.135181 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.135099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e977f591-eb58-419c-9afe-5f391cf1d8e4-kserve-provision-location\") pod \"isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") " pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.135181 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.135126    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e977f591-eb58-419c-9afe-5f391cf1d8e4-proxy-tls\") pod \"isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") " pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.135181 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.135173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e977f591-eb58-419c-9afe-5f391cf1d8e4-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") " pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.135286 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.135201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-4h7qs\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.135286 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.135235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-4h7qs\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.135286 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.135266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-4h7qs\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.135407 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.135292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-4h7qs\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.135407 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.135326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrg7j\" (UniqueName: \"kubernetes.io/projected/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kube-api-access-lrg7j\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-4h7qs\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.135407 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.135363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e977f591-eb58-419c-9afe-5f391cf1d8e4-cabundle-cert\") pod \"isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") " pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.135609 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.135553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-4h7qs\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.136023 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.136004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-4h7qs\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.136091 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.136027    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-4h7qs\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.137724 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.137710    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-4h7qs\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.144658 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.144635    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrg7j\" (UniqueName: \"kubernetes.io/projected/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kube-api-access-lrg7j\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-4h7qs\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.235773 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.235748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e977f591-eb58-419c-9afe-5f391cf1d8e4-cabundle-cert\") pod \"isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") " pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.235866 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.235788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqnd\" (UniqueName: \"kubernetes.io/projected/e977f591-eb58-419c-9afe-5f391cf1d8e4-kube-api-access-btqnd\") pod \"isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") " pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.235866 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.235807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e977f591-eb58-419c-9afe-5f391cf1d8e4-kserve-provision-location\") pod \"isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") " pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.235866 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.235822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e977f591-eb58-419c-9afe-5f391cf1d8e4-proxy-tls\") pod \"isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") " pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.236009 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.235937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e977f591-eb58-419c-9afe-5f391cf1d8e4-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") " pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.236219 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.236198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e977f591-eb58-419c-9afe-5f391cf1d8e4-kserve-provision-location\") pod \"isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") " pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.236426 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.236409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e977f591-eb58-419c-9afe-5f391cf1d8e4-cabundle-cert\") pod \"isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") " pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.236519 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.236500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e977f591-eb58-419c-9afe-5f391cf1d8e4-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") " pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.237938 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.237917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e977f591-eb58-419c-9afe-5f391cf1d8e4-proxy-tls\") pod \"isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") " pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.246185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.246170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btqnd\" (UniqueName: \"kubernetes.io/projected/e977f591-eb58-419c-9afe-5f391cf1d8e4-kube-api-access-btqnd\") pod \"isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") " pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.246271 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.246260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:02.270906 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.270875    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:02.373393 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.373324    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:02.519350 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.519322    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"]
Sep 16 17:38:02.521980 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:38:02.521953    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9ae155_6e6d_46d9_bf38_d350dde1c93f.slice/crio-8efceef30e26a83073d0e66a00547a5d7cb0272304fa71051738f4ea1473e1cd WatchSource:0}: Error finding container 8efceef30e26a83073d0e66a00547a5d7cb0272304fa71051738f4ea1473e1cd: Status 404 returned error can't find the container with id 8efceef30e26a83073d0e66a00547a5d7cb0272304fa71051738f4ea1473e1cd
Sep 16 17:38:02.644236 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.644213    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"]
Sep 16 17:38:02.646389 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:38:02.646365    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode977f591_eb58_419c_9afe_5f391cf1d8e4.slice/crio-2966b1a569fe4573bcb5e30b13ad33c98e4b7771609f1e9dc3773fe8ccb0e2fb WatchSource:0}: Error finding container 2966b1a569fe4573bcb5e30b13ad33c98e4b7771609f1e9dc3773fe8ccb0e2fb: Status 404 returned error can't find the container with id 2966b1a569fe4573bcb5e30b13ad33c98e4b7771609f1e9dc3773fe8ccb0e2fb
Sep 16 17:38:02.916142 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.916056    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh" event={"ID":"e977f591-eb58-419c-9afe-5f391cf1d8e4","Type":"ContainerStarted","Data":"5cdf3dc01850e4dcdb5d5548c84e364626453845162b74419a6d7ab619b5c57d"}
Sep 16 17:38:02.916142 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.916122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh" event={"ID":"e977f591-eb58-419c-9afe-5f391cf1d8e4","Type":"ContainerStarted","Data":"2966b1a569fe4573bcb5e30b13ad33c98e4b7771609f1e9dc3773fe8ccb0e2fb"}
Sep 16 17:38:02.917753 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.917718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" event={"ID":"8b9ae155-6e6d-46d9-bf38-d350dde1c93f","Type":"ContainerStarted","Data":"8b3b80eebe895c447c3323c2e1f0ee868f766eb60055809d1968149593f6d857"}
Sep 16 17:38:02.917874 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:02.917795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" event={"ID":"8b9ae155-6e6d-46d9-bf38-d350dde1c93f","Type":"ContainerStarted","Data":"8efceef30e26a83073d0e66a00547a5d7cb0272304fa71051738f4ea1473e1cd"}
Sep 16 17:38:04.928782 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:04.928735    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh_e977f591-eb58-419c-9afe-5f391cf1d8e4/storage-initializer/0.log"
Sep 16 17:38:04.929096 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:04.928776    2581 generic.go:356] "Generic (PLEG): container finished" podID="e977f591-eb58-419c-9afe-5f391cf1d8e4" containerID="5cdf3dc01850e4dcdb5d5548c84e364626453845162b74419a6d7ab619b5c57d" exitCode=1
Sep 16 17:38:04.929096 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:04.928857    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh" event={"ID":"e977f591-eb58-419c-9afe-5f391cf1d8e4","Type":"ContainerDied","Data":"5cdf3dc01850e4dcdb5d5548c84e364626453845162b74419a6d7ab619b5c57d"}
Sep 16 17:38:05.935028 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:05.935000    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh_e977f591-eb58-419c-9afe-5f391cf1d8e4/storage-initializer/0.log"
Sep 16 17:38:05.935395 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:05.935113    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh" event={"ID":"e977f591-eb58-419c-9afe-5f391cf1d8e4","Type":"ContainerStarted","Data":"2ca0b1722a2c1055ec42b8191c8c2db49181ce371390f74aaa6bfcb35a6a9924"}
Sep 16 17:38:06.691379 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:06.691349    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"]
Sep 16 17:38:06.702154 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:06.702124    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:06.941004 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:06.940979    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerID="8b3b80eebe895c447c3323c2e1f0ee868f766eb60055809d1968149593f6d857" exitCode=0
Sep 16 17:38:06.941311 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:06.941056    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" event={"ID":"8b9ae155-6e6d-46d9-bf38-d350dde1c93f","Type":"ContainerDied","Data":"8b3b80eebe895c447c3323c2e1f0ee868f766eb60055809d1968149593f6d857"}
Sep 16 17:38:06.941357 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:06.941329    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh" podUID="e977f591-eb58-419c-9afe-5f391cf1d8e4" containerName="storage-initializer" containerID="cri-o://2ca0b1722a2c1055ec42b8191c8c2db49181ce371390f74aaa6bfcb35a6a9924" gracePeriod=30
Sep 16 17:38:11.629355 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.629333    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh_e977f591-eb58-419c-9afe-5f391cf1d8e4/storage-initializer/1.log"
Sep 16 17:38:11.629699 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.629683    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh_e977f591-eb58-419c-9afe-5f391cf1d8e4/storage-initializer/0.log"
Sep 16 17:38:11.632049 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.632014    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:11.721175 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.721152    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e977f591-eb58-419c-9afe-5f391cf1d8e4-proxy-tls\") pod \"e977f591-eb58-419c-9afe-5f391cf1d8e4\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") "
Sep 16 17:38:11.721289 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.721186    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btqnd\" (UniqueName: \"kubernetes.io/projected/e977f591-eb58-419c-9afe-5f391cf1d8e4-kube-api-access-btqnd\") pod \"e977f591-eb58-419c-9afe-5f391cf1d8e4\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") "
Sep 16 17:38:11.721289 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.721217    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e977f591-eb58-419c-9afe-5f391cf1d8e4-cabundle-cert\") pod \"e977f591-eb58-419c-9afe-5f391cf1d8e4\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") "
Sep 16 17:38:11.721289 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.721243    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e977f591-eb58-419c-9afe-5f391cf1d8e4-kserve-provision-location\") pod \"e977f591-eb58-419c-9afe-5f391cf1d8e4\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") "
Sep 16 17:38:11.721289 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.721266    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e977f591-eb58-419c-9afe-5f391cf1d8e4-kube-rbac-proxy-sar-config\") pod \"e977f591-eb58-419c-9afe-5f391cf1d8e4\" (UID: \"e977f591-eb58-419c-9afe-5f391cf1d8e4\") "
Sep 16 17:38:11.721527 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.721506    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e977f591-eb58-419c-9afe-5f391cf1d8e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e977f591-eb58-419c-9afe-5f391cf1d8e4" (UID: "e977f591-eb58-419c-9afe-5f391cf1d8e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:38:11.721637 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.721610    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e977f591-eb58-419c-9afe-5f391cf1d8e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e977f591-eb58-419c-9afe-5f391cf1d8e4" (UID: "e977f591-eb58-419c-9afe-5f391cf1d8e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:38:11.721728 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.721690    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e977f591-eb58-419c-9afe-5f391cf1d8e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e977f591-eb58-419c-9afe-5f391cf1d8e4" (UID: "e977f591-eb58-419c-9afe-5f391cf1d8e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:38:11.723190 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.723151    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e977f591-eb58-419c-9afe-5f391cf1d8e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e977f591-eb58-419c-9afe-5f391cf1d8e4" (UID: "e977f591-eb58-419c-9afe-5f391cf1d8e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:38:11.723301 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.723286    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e977f591-eb58-419c-9afe-5f391cf1d8e4-kube-api-access-btqnd" (OuterVolumeSpecName: "kube-api-access-btqnd") pod "e977f591-eb58-419c-9afe-5f391cf1d8e4" (UID: "e977f591-eb58-419c-9afe-5f391cf1d8e4"). InnerVolumeSpecName "kube-api-access-btqnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:38:11.821988 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.821962    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e977f591-eb58-419c-9afe-5f391cf1d8e4-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:38:11.822108 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.821989    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btqnd\" (UniqueName: \"kubernetes.io/projected/e977f591-eb58-419c-9afe-5f391cf1d8e4-kube-api-access-btqnd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:38:11.822108 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.822021    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e977f591-eb58-419c-9afe-5f391cf1d8e4-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:38:11.822108 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.822033    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e977f591-eb58-419c-9afe-5f391cf1d8e4-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:38:11.822108 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.822065    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e977f591-eb58-419c-9afe-5f391cf1d8e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:38:11.822297 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.822156    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:11.865051 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.865024    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"]
Sep 16 17:38:11.865383 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.865367    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e977f591-eb58-419c-9afe-5f391cf1d8e4" containerName="storage-initializer"
Sep 16 17:38:11.865383 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.865383    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e977f591-eb58-419c-9afe-5f391cf1d8e4" containerName="storage-initializer"
Sep 16 17:38:11.865521 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.865414    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e977f591-eb58-419c-9afe-5f391cf1d8e4" containerName="storage-initializer"
Sep 16 17:38:11.865521 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.865420    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e977f591-eb58-419c-9afe-5f391cf1d8e4" containerName="storage-initializer"
Sep 16 17:38:11.865521 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.865486    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e977f591-eb58-419c-9afe-5f391cf1d8e4" containerName="storage-initializer"
Sep 16 17:38:11.865521 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.865494    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e977f591-eb58-419c-9afe-5f391cf1d8e4" containerName="storage-initializer"
Sep 16 17:38:11.868861 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.868842    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:11.871712 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.871693    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 16 17:38:11.871812 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.871801    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 16 17:38:11.893116 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.893095    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"]
Sep 16 17:38:11.970731 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.970706    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh_e977f591-eb58-419c-9afe-5f391cf1d8e4/storage-initializer/1.log"
Sep 16 17:38:11.971153 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.971137    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh_e977f591-eb58-419c-9afe-5f391cf1d8e4/storage-initializer/0.log"
Sep 16 17:38:11.971259 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.971178    2581 generic.go:356] "Generic (PLEG): container finished" podID="e977f591-eb58-419c-9afe-5f391cf1d8e4" containerID="2ca0b1722a2c1055ec42b8191c8c2db49181ce371390f74aaa6bfcb35a6a9924" exitCode=1
Sep 16 17:38:11.971259 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.971222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh" event={"ID":"e977f591-eb58-419c-9afe-5f391cf1d8e4","Type":"ContainerDied","Data":"2ca0b1722a2c1055ec42b8191c8c2db49181ce371390f74aaa6bfcb35a6a9924"}
Sep 16 17:38:11.971361 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.971259    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"
Sep 16 17:38:11.971361 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.971276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh" event={"ID":"e977f591-eb58-419c-9afe-5f391cf1d8e4","Type":"ContainerDied","Data":"2966b1a569fe4573bcb5e30b13ad33c98e4b7771609f1e9dc3773fe8ccb0e2fb"}
Sep 16 17:38:11.971361 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.971315    2581 scope.go:117] "RemoveContainer" containerID="2ca0b1722a2c1055ec42b8191c8c2db49181ce371390f74aaa6bfcb35a6a9924"
Sep 16 17:38:11.980602 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.980584    2581 scope.go:117] "RemoveContainer" containerID="5cdf3dc01850e4dcdb5d5548c84e364626453845162b74419a6d7ab619b5c57d"
Sep 16 17:38:11.989982 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.989952    2581 scope.go:117] "RemoveContainer" containerID="2ca0b1722a2c1055ec42b8191c8c2db49181ce371390f74aaa6bfcb35a6a9924"
Sep 16 17:38:11.990250 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:38:11.990226    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca0b1722a2c1055ec42b8191c8c2db49181ce371390f74aaa6bfcb35a6a9924\": container with ID starting with 2ca0b1722a2c1055ec42b8191c8c2db49181ce371390f74aaa6bfcb35a6a9924 not found: ID does not exist" containerID="2ca0b1722a2c1055ec42b8191c8c2db49181ce371390f74aaa6bfcb35a6a9924"
Sep 16 17:38:11.990328 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.990260    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca0b1722a2c1055ec42b8191c8c2db49181ce371390f74aaa6bfcb35a6a9924"} err="failed to get container status \"2ca0b1722a2c1055ec42b8191c8c2db49181ce371390f74aaa6bfcb35a6a9924\": rpc error: code = NotFound desc = could not find container \"2ca0b1722a2c1055ec42b8191c8c2db49181ce371390f74aaa6bfcb35a6a9924\": container with ID starting with 2ca0b1722a2c1055ec42b8191c8c2db49181ce371390f74aaa6bfcb35a6a9924 not found: ID does not exist"
Sep 16 17:38:11.990328 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.990283    2581 scope.go:117] "RemoveContainer" containerID="5cdf3dc01850e4dcdb5d5548c84e364626453845162b74419a6d7ab619b5c57d"
Sep 16 17:38:11.990586 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:38:11.990567    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cdf3dc01850e4dcdb5d5548c84e364626453845162b74419a6d7ab619b5c57d\": container with ID starting with 5cdf3dc01850e4dcdb5d5548c84e364626453845162b74419a6d7ab619b5c57d not found: ID does not exist" containerID="5cdf3dc01850e4dcdb5d5548c84e364626453845162b74419a6d7ab619b5c57d"
Sep 16 17:38:11.990660 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:11.990594    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cdf3dc01850e4dcdb5d5548c84e364626453845162b74419a6d7ab619b5c57d"} err="failed to get container status \"5cdf3dc01850e4dcdb5d5548c84e364626453845162b74419a6d7ab619b5c57d\": rpc error: code = NotFound desc = could not find container \"5cdf3dc01850e4dcdb5d5548c84e364626453845162b74419a6d7ab619b5c57d\": container with ID starting with 5cdf3dc01850e4dcdb5d5548c84e364626453845162b74419a6d7ab619b5c57d not found: ID does not exist"
Sep 16 17:38:12.024327 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.024302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b121d39d-642c-41ef-be51-f261c23a9ad8-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-btrjg\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:12.024442 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.024383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xm7\" (UniqueName: \"kubernetes.io/projected/b121d39d-642c-41ef-be51-f261c23a9ad8-kube-api-access-f4xm7\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-btrjg\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:12.024442 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.024419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b121d39d-642c-41ef-be51-f261c23a9ad8-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-btrjg\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:12.024547 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.024455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b121d39d-642c-41ef-be51-f261c23a9ad8-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-btrjg\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:12.024547 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.024484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b121d39d-642c-41ef-be51-f261c23a9ad8-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-btrjg\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:12.027512 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.027486    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"]
Sep 16 17:38:12.031655 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.031636    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a427f5-predictor-575ffbbd88-29qrh"]
Sep 16 17:38:12.129480 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.128927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b121d39d-642c-41ef-be51-f261c23a9ad8-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-btrjg\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:12.129480 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.128983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b121d39d-642c-41ef-be51-f261c23a9ad8-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-btrjg\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:12.129480 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.129062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b121d39d-642c-41ef-be51-f261c23a9ad8-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-btrjg\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:12.129480 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.129135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xm7\" (UniqueName: \"kubernetes.io/projected/b121d39d-642c-41ef-be51-f261c23a9ad8-kube-api-access-f4xm7\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-btrjg\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:12.129480 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.129177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b121d39d-642c-41ef-be51-f261c23a9ad8-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-btrjg\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:12.130072 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.129790    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b121d39d-642c-41ef-be51-f261c23a9ad8-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-btrjg\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:12.130072 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.130078    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b121d39d-642c-41ef-be51-f261c23a9ad8-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-btrjg\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:12.130356 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.130132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b121d39d-642c-41ef-be51-f261c23a9ad8-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-btrjg\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:12.132494 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.132466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b121d39d-642c-41ef-be51-f261c23a9ad8-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-btrjg\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:12.141634 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.141609    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xm7\" (UniqueName: \"kubernetes.io/projected/b121d39d-642c-41ef-be51-f261c23a9ad8-kube-api-access-f4xm7\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-btrjg\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:12.172747 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.172721    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:12.181115 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:12.181093    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:13.767095 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:13.767060    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e977f591-eb58-419c-9afe-5f391cf1d8e4" path="/var/lib/kubelet/pods/e977f591-eb58-419c-9afe-5f391cf1d8e4/volumes"
Sep 16 17:38:19.702723 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:19.702667    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"]
Sep 16 17:38:19.705359 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:38:19.705327    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb121d39d_642c_41ef_be51_f261c23a9ad8.slice/crio-376e35db1b9808957a743e35f44cdc58d0b0c4eb9ce2c574ea436f39375267e4 WatchSource:0}: Error finding container 376e35db1b9808957a743e35f44cdc58d0b0c4eb9ce2c574ea436f39375267e4: Status 404 returned error can't find the container with id 376e35db1b9808957a743e35f44cdc58d0b0c4eb9ce2c574ea436f39375267e4
Sep 16 17:38:20.014117 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:20.014035    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" event={"ID":"8b9ae155-6e6d-46d9-bf38-d350dde1c93f","Type":"ContainerStarted","Data":"2d51581073d960fd3d1fcc7f3b4e333053ca1f19753c08d3a01c71783acc58a9"}
Sep 16 17:38:20.015680 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:20.015643    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" event={"ID":"b121d39d-642c-41ef-be51-f261c23a9ad8","Type":"ContainerStarted","Data":"ffc88bf4e9bff60094fae209d765b911a9d9897cbc9fd3909113faac1d9ab8d6"}
Sep 16 17:38:20.015825 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:20.015692    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" event={"ID":"b121d39d-642c-41ef-be51-f261c23a9ad8","Type":"ContainerStarted","Data":"376e35db1b9808957a743e35f44cdc58d0b0c4eb9ce2c574ea436f39375267e4"}
Sep 16 17:38:20.253614 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:20.253579    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:23.030701 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:23.030670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" event={"ID":"8b9ae155-6e6d-46d9-bf38-d350dde1c93f","Type":"ContainerStarted","Data":"8af4349fb4e602a238d6210d3b15e6dd9dca0d400a6a7a5ab3c8805c079106d7"}
Sep 16 17:38:25.041872 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:25.041833    2581 generic.go:356] "Generic (PLEG): container finished" podID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerID="ffc88bf4e9bff60094fae209d765b911a9d9897cbc9fd3909113faac1d9ab8d6" exitCode=0
Sep 16 17:38:25.041872 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:25.041879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" event={"ID":"b121d39d-642c-41ef-be51-f261c23a9ad8","Type":"ContainerDied","Data":"ffc88bf4e9bff60094fae209d765b911a9d9897cbc9fd3909113faac1d9ab8d6"}
Sep 16 17:38:26.056629 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:26.056594    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" event={"ID":"8b9ae155-6e6d-46d9-bf38-d350dde1c93f","Type":"ContainerStarted","Data":"cee606cd85f6d93de65f0f20fd1f28306504257db0d13e2ea26c8cdea1a92f19"}
Sep 16 17:38:26.057960 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:26.057789    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:26.057960 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:26.057852    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:26.057960 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:26.057907    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:26.059801 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:26.059561    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 17:38:26.060236 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:26.060161    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:38:26.096751 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:26.096196    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podStartSLOduration=3.115704841 podStartE2EDuration="25.096177932s" totalImagesPullingTime="18.042214084s" totalInitContainerRuntime="3.938259007s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:38:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:38:26.094330283 +0000 UTC m=+834.882952817" watchObservedRunningTime="2026-09-16 17:38:26.096177932 +0000 UTC m=+834.884800464"
Sep 16 17:38:27.064615 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:27.064575    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 17:38:27.065662 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:27.065597    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:38:27.073001 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:27.072953    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:38:28.072087 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:28.072032    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 17:38:28.072503 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:28.072392    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:38:29.081696 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:29.081332    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 17:38:29.082316 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:29.082258    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:38:30.724463 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:30.724432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:39.081748 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:39.081697    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 17:38:39.082231 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:39.082139    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:38:48.179354 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:48.179319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" event={"ID":"b121d39d-642c-41ef-be51-f261c23a9ad8","Type":"ContainerStarted","Data":"049435033a8e734bbebd42d34c10e62a16891613a0cbfe7e477612ebbe1f1b18"}
Sep 16 17:38:48.179722 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:48.179379    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" event={"ID":"b121d39d-642c-41ef-be51-f261c23a9ad8","Type":"ContainerStarted","Data":"8ac6d46f83411dc83f00a746e687a33910d5b4dd48fe7d0b2e2e713bb891a64e"}
Sep 16 17:38:48.179722 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:48.179552    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:49.081668 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:49.081619    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 17:38:49.082171 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:49.082146    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:38:49.183287 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:49.183254    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:49.184556 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:49.184532    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 17:38:50.187010 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:50.186976    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 17:38:55.191095 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:55.191066    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:38:55.191578 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:55.191554    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 17:38:55.218955 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:55.218909    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" podStartSLOduration=17.450550571 podStartE2EDuration="44.218879198s" totalImagesPullingTime="22.505342507s" totalInitContainerRuntime="4.26298612s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:38:48.202885079 +0000 UTC m=+856.991507611" watchObservedRunningTime="2026-09-16 17:38:55.218879198 +0000 UTC m=+864.007501788"
Sep 16 17:38:59.081203 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:59.081155    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 17:38:59.081662 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:38:59.081638    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:39:05.191774 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:05.191739    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 17:39:09.082088 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:09.082043    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 17:39:09.082526 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:09.082504    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:39:15.191633 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:15.191598    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 17:39:19.081667 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:19.081626    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 17:39:19.082280 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:19.082254    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:39:25.191683 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:25.191604    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 17:39:29.081593 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:29.081548    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 17:39:29.082089 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:29.082040    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:39:31.709249 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:31.709221    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:39:31.712634 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:31.712610    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:39:35.192343 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:35.192306    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 17:39:39.082086 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:39.082049    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:39:39.082495 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:39.082481    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:39:45.192327 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:45.192296    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 17:39:46.907432 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:46.907403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:39:46.996446 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:46.996410    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"]
Sep 16 17:39:46.996722 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:46.996693    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container" containerID="cri-o://2d51581073d960fd3d1fcc7f3b4e333053ca1f19753c08d3a01c71783acc58a9" gracePeriod=30
Sep 16 17:39:46.996839 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:46.996713    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent" containerID="cri-o://cee606cd85f6d93de65f0f20fd1f28306504257db0d13e2ea26c8cdea1a92f19" gracePeriod=30
Sep 16 17:39:46.996839 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:46.996766    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kube-rbac-proxy" containerID="cri-o://8af4349fb4e602a238d6210d3b15e6dd9dca0d400a6a7a5ab3c8805c079106d7" gracePeriod=30
Sep 16 17:39:47.011111 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.011093    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:39:47.027123 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.027104    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"]
Sep 16 17:39:47.030717 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.030703    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.034552 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.034530    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 16 17:39:47.034638 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.034600    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 16 17:39:47.065082 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.065060    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 16 17:39:47.065342 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.065323    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"]
Sep 16 17:39:47.074605 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.074583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.074687 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.074613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcqfw\" (UniqueName: \"kubernetes.io/projected/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kube-api-access-vcqfw\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.074687 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.074658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb9decc-819c-4c22-94b6-0b0e37d1efb2-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.074791 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.074763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edb9decc-819c-4c22-94b6-0b0e37d1efb2-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.074875 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.074829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.147116 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.147098    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:39:47.175339 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.175295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.175339 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.175320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcqfw\" (UniqueName: \"kubernetes.io/projected/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kube-api-access-vcqfw\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.175485 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.175343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb9decc-819c-4c22-94b6-0b0e37d1efb2-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.175541 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.175481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edb9decc-819c-4c22-94b6-0b0e37d1efb2-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.175593 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.175539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.175880 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.175859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.176010 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.175998    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.176081 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.176062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edb9decc-819c-4c22-94b6-0b0e37d1efb2-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.177781 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.177765    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb9decc-819c-4c22-94b6-0b0e37d1efb2-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.193313 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.193291    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcqfw\" (UniqueName: \"kubernetes.io/projected/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kube-api-access-vcqfw\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.339662 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.339639    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:47.423853 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.423827    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerID="8af4349fb4e602a238d6210d3b15e6dd9dca0d400a6a7a5ab3c8805c079106d7" exitCode=2
Sep 16 17:39:47.424002 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.423906    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" event={"ID":"8b9ae155-6e6d-46d9-bf38-d350dde1c93f","Type":"ContainerDied","Data":"8af4349fb4e602a238d6210d3b15e6dd9dca0d400a6a7a5ab3c8805c079106d7"}
Sep 16 17:39:47.625782 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:47.625757    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"]
Sep 16 17:39:47.628225 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:39:47.628201    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb9decc_819c_4c22_94b6_0b0e37d1efb2.slice/crio-2280d4c0d7317c953ab8939b51459c00fae2fb2288c5e2bbb219e802418b6e50 WatchSource:0}: Error finding container 2280d4c0d7317c953ab8939b51459c00fae2fb2288c5e2bbb219e802418b6e50: Status 404 returned error can't find the container with id 2280d4c0d7317c953ab8939b51459c00fae2fb2288c5e2bbb219e802418b6e50
Sep 16 17:39:48.430392 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:48.430358    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" event={"ID":"edb9decc-819c-4c22-94b6-0b0e37d1efb2","Type":"ContainerStarted","Data":"1e90afaa6a6ab706def46b93813d42b3c4875c3e45c625074d7a0462f278a2b1"}
Sep 16 17:39:48.430750 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:48.430422    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" event={"ID":"edb9decc-819c-4c22-94b6-0b0e37d1efb2","Type":"ContainerStarted","Data":"2280d4c0d7317c953ab8939b51459c00fae2fb2288c5e2bbb219e802418b6e50"}
Sep 16 17:39:49.081839 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:49.081804    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 17:39:49.083367 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:49.083338    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:39:51.446517 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:51.446482    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerID="2d51581073d960fd3d1fcc7f3b4e333053ca1f19753c08d3a01c71783acc58a9" exitCode=0
Sep 16 17:39:51.446862 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:51.446554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" event={"ID":"8b9ae155-6e6d-46d9-bf38-d350dde1c93f","Type":"ContainerDied","Data":"2d51581073d960fd3d1fcc7f3b4e333053ca1f19753c08d3a01c71783acc58a9"}
Sep 16 17:39:51.448100 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:51.448079    2581 generic.go:356] "Generic (PLEG): container finished" podID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerID="1e90afaa6a6ab706def46b93813d42b3c4875c3e45c625074d7a0462f278a2b1" exitCode=0
Sep 16 17:39:51.448219 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:51.448115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" event={"ID":"edb9decc-819c-4c22-94b6-0b0e37d1efb2","Type":"ContainerDied","Data":"1e90afaa6a6ab706def46b93813d42b3c4875c3e45c625074d7a0462f278a2b1"}
Sep 16 17:39:52.064820 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:52.064780    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 16 17:39:52.455192 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:52.455116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" event={"ID":"edb9decc-819c-4c22-94b6-0b0e37d1efb2","Type":"ContainerStarted","Data":"e1cb4dd8eea3710998f077954e5a5ae779b1c8bb1d96720f6bff305d9821d247"}
Sep 16 17:39:52.455570 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:52.455194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" event={"ID":"edb9decc-819c-4c22-94b6-0b0e37d1efb2","Type":"ContainerStarted","Data":"f88d5b61d38ac2332ff474fc37baa926e727d84cd91fe83af80499135d5dc8d1"}
Sep 16 17:39:52.455570 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:52.455227    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" event={"ID":"edb9decc-819c-4c22-94b6-0b0e37d1efb2","Type":"ContainerStarted","Data":"0814f158b27db7b25bad18757b0e1f27458a433024e562810650a41d9dd31bd9"}
Sep 16 17:39:52.455570 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:52.455516    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:52.455570 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:52.455563    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:52.455792 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:52.455585    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:52.456949 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:52.456913    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:5000: connect: connection refused"
Sep 16 17:39:52.457507 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:52.457484    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:39:52.481101 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:52.481049    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podStartSLOduration=2.763449095 podStartE2EDuration="6.481038227s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.717589132s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:39:52.47895733 +0000 UTC m=+921.267579863" watchObservedRunningTime="2026-09-16 17:39:52.481038227 +0000 UTC m=+921.269660758"
Sep 16 17:39:53.460119 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:53.460080    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:5000: connect: connection refused"
Sep 16 17:39:53.460474 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:53.460400    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:39:55.192055 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:55.192026    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:39:57.064911 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:57.064849    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 16 17:39:57.065319 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:57.064981    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:39:58.464070 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:58.464040    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:39:58.464644 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:58.464615    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:5000: connect: connection refused"
Sep 16 17:39:58.464927 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:58.464869    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:39:59.081565 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:59.081534    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 17:39:59.083099 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:39:59.083075    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:40:02.013634 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:02.013604    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"]
Sep 16 17:40:02.013991 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:02.013846    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kserve-container" containerID="cri-o://8ac6d46f83411dc83f00a746e687a33910d5b4dd48fe7d0b2e2e713bb891a64e" gracePeriod=30
Sep 16 17:40:02.013991 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:02.013915    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kube-rbac-proxy" containerID="cri-o://049435033a8e734bbebd42d34c10e62a16891613a0cbfe7e477612ebbe1f1b18" gracePeriod=30
Sep 16 17:40:02.023719 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:02.023693    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:40:02.065585 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:02.065557    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 16 17:40:02.501529 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:02.501504    2581 generic.go:356] "Generic (PLEG): container finished" podID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerID="049435033a8e734bbebd42d34c10e62a16891613a0cbfe7e477612ebbe1f1b18" exitCode=2
Sep 16 17:40:02.501655 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:02.501575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" event={"ID":"b121d39d-642c-41ef-be51-f261c23a9ad8","Type":"ContainerDied","Data":"049435033a8e734bbebd42d34c10e62a16891613a0cbfe7e477612ebbe1f1b18"}
Sep 16 17:40:05.187306 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.187269    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 16 17:40:05.191615 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.191592    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 17:40:05.646070 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.646048    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:40:05.712827 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.712803    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b121d39d-642c-41ef-be51-f261c23a9ad8-kserve-provision-location\") pod \"b121d39d-642c-41ef-be51-f261c23a9ad8\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") "
Sep 16 17:40:05.712959 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.712840    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b121d39d-642c-41ef-be51-f261c23a9ad8-kube-rbac-proxy-sar-config\") pod \"b121d39d-642c-41ef-be51-f261c23a9ad8\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") "
Sep 16 17:40:05.712959 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.712883    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b121d39d-642c-41ef-be51-f261c23a9ad8-cabundle-cert\") pod \"b121d39d-642c-41ef-be51-f261c23a9ad8\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") "
Sep 16 17:40:05.713071 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.713043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4xm7\" (UniqueName: \"kubernetes.io/projected/b121d39d-642c-41ef-be51-f261c23a9ad8-kube-api-access-f4xm7\") pod \"b121d39d-642c-41ef-be51-f261c23a9ad8\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") "
Sep 16 17:40:05.713187 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.713097    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b121d39d-642c-41ef-be51-f261c23a9ad8-proxy-tls\") pod \"b121d39d-642c-41ef-be51-f261c23a9ad8\" (UID: \"b121d39d-642c-41ef-be51-f261c23a9ad8\") "
Sep 16 17:40:05.713187 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.713144    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b121d39d-642c-41ef-be51-f261c23a9ad8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b121d39d-642c-41ef-be51-f261c23a9ad8" (UID: "b121d39d-642c-41ef-be51-f261c23a9ad8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:40:05.713289 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.713219    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b121d39d-642c-41ef-be51-f261c23a9ad8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b121d39d-642c-41ef-be51-f261c23a9ad8" (UID: "b121d39d-642c-41ef-be51-f261c23a9ad8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:40:05.713289 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.713250    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b121d39d-642c-41ef-be51-f261c23a9ad8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b121d39d-642c-41ef-be51-f261c23a9ad8" (UID: "b121d39d-642c-41ef-be51-f261c23a9ad8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:40:05.713370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.713338    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b121d39d-642c-41ef-be51-f261c23a9ad8-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:40:05.713370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.713352    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b121d39d-642c-41ef-be51-f261c23a9ad8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:40:05.713370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.713362    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b121d39d-642c-41ef-be51-f261c23a9ad8-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:40:05.715020 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.714995    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b121d39d-642c-41ef-be51-f261c23a9ad8-kube-api-access-f4xm7" (OuterVolumeSpecName: "kube-api-access-f4xm7") pod "b121d39d-642c-41ef-be51-f261c23a9ad8" (UID: "b121d39d-642c-41ef-be51-f261c23a9ad8"). InnerVolumeSpecName "kube-api-access-f4xm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:40:05.715098 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.715049    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b121d39d-642c-41ef-be51-f261c23a9ad8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b121d39d-642c-41ef-be51-f261c23a9ad8" (UID: "b121d39d-642c-41ef-be51-f261c23a9ad8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:40:05.814643 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.814618    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4xm7\" (UniqueName: \"kubernetes.io/projected/b121d39d-642c-41ef-be51-f261c23a9ad8-kube-api-access-f4xm7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:40:05.814751 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:05.814643    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b121d39d-642c-41ef-be51-f261c23a9ad8-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:40:06.521104 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:06.521076    2581 generic.go:356] "Generic (PLEG): container finished" podID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerID="8ac6d46f83411dc83f00a746e687a33910d5b4dd48fe7d0b2e2e713bb891a64e" exitCode=0
Sep 16 17:40:06.521436 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:06.521128    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"
Sep 16 17:40:06.521436 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:06.521154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" event={"ID":"b121d39d-642c-41ef-be51-f261c23a9ad8","Type":"ContainerDied","Data":"8ac6d46f83411dc83f00a746e687a33910d5b4dd48fe7d0b2e2e713bb891a64e"}
Sep 16 17:40:06.521436 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:06.521233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg" event={"ID":"b121d39d-642c-41ef-be51-f261c23a9ad8","Type":"ContainerDied","Data":"376e35db1b9808957a743e35f44cdc58d0b0c4eb9ce2c574ea436f39375267e4"}
Sep 16 17:40:06.521436 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:06.521270    2581 scope.go:117] "RemoveContainer" containerID="049435033a8e734bbebd42d34c10e62a16891613a0cbfe7e477612ebbe1f1b18"
Sep 16 17:40:06.529044 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:06.529020    2581 scope.go:117] "RemoveContainer" containerID="8ac6d46f83411dc83f00a746e687a33910d5b4dd48fe7d0b2e2e713bb891a64e"
Sep 16 17:40:06.536312 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:06.536298    2581 scope.go:117] "RemoveContainer" containerID="ffc88bf4e9bff60094fae209d765b911a9d9897cbc9fd3909113faac1d9ab8d6"
Sep 16 17:40:06.543414 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:06.543393    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"]
Sep 16 17:40:06.543763 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:06.543743    2581 scope.go:117] "RemoveContainer" containerID="049435033a8e734bbebd42d34c10e62a16891613a0cbfe7e477612ebbe1f1b18"
Sep 16 17:40:06.544058 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:40:06.544036    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"049435033a8e734bbebd42d34c10e62a16891613a0cbfe7e477612ebbe1f1b18\": container with ID starting with 049435033a8e734bbebd42d34c10e62a16891613a0cbfe7e477612ebbe1f1b18 not found: ID does not exist" containerID="049435033a8e734bbebd42d34c10e62a16891613a0cbfe7e477612ebbe1f1b18"
Sep 16 17:40:06.544118 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:06.544067    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"049435033a8e734bbebd42d34c10e62a16891613a0cbfe7e477612ebbe1f1b18"} err="failed to get container status \"049435033a8e734bbebd42d34c10e62a16891613a0cbfe7e477612ebbe1f1b18\": rpc error: code = NotFound desc = could not find container \"049435033a8e734bbebd42d34c10e62a16891613a0cbfe7e477612ebbe1f1b18\": container with ID starting with 049435033a8e734bbebd42d34c10e62a16891613a0cbfe7e477612ebbe1f1b18 not found: ID does not exist"
Sep 16 17:40:06.544118 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:06.544082    2581 scope.go:117] "RemoveContainer" containerID="8ac6d46f83411dc83f00a746e687a33910d5b4dd48fe7d0b2e2e713bb891a64e"
Sep 16 17:40:06.544327 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:40:06.544311    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac6d46f83411dc83f00a746e687a33910d5b4dd48fe7d0b2e2e713bb891a64e\": container with ID starting with 8ac6d46f83411dc83f00a746e687a33910d5b4dd48fe7d0b2e2e713bb891a64e not found: ID does not exist" containerID="8ac6d46f83411dc83f00a746e687a33910d5b4dd48fe7d0b2e2e713bb891a64e"
Sep 16 17:40:06.544369 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:06.544333    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac6d46f83411dc83f00a746e687a33910d5b4dd48fe7d0b2e2e713bb891a64e"} err="failed to get container status \"8ac6d46f83411dc83f00a746e687a33910d5b4dd48fe7d0b2e2e713bb891a64e\": rpc error: code = NotFound desc = could not find container \"8ac6d46f83411dc83f00a746e687a33910d5b4dd48fe7d0b2e2e713bb891a64e\": container with ID starting with 8ac6d46f83411dc83f00a746e687a33910d5b4dd48fe7d0b2e2e713bb891a64e not found: ID does not exist"
Sep 16 17:40:06.544369 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:06.544347    2581 scope.go:117] "RemoveContainer" containerID="ffc88bf4e9bff60094fae209d765b911a9d9897cbc9fd3909113faac1d9ab8d6"
Sep 16 17:40:06.544585 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:40:06.544570    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffc88bf4e9bff60094fae209d765b911a9d9897cbc9fd3909113faac1d9ab8d6\": container with ID starting with ffc88bf4e9bff60094fae209d765b911a9d9897cbc9fd3909113faac1d9ab8d6 not found: ID does not exist" containerID="ffc88bf4e9bff60094fae209d765b911a9d9897cbc9fd3909113faac1d9ab8d6"
Sep 16 17:40:06.544623 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:06.544589    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffc88bf4e9bff60094fae209d765b911a9d9897cbc9fd3909113faac1d9ab8d6"} err="failed to get container status \"ffc88bf4e9bff60094fae209d765b911a9d9897cbc9fd3909113faac1d9ab8d6\": rpc error: code = NotFound desc = could not find container \"ffc88bf4e9bff60094fae209d765b911a9d9897cbc9fd3909113faac1d9ab8d6\": container with ID starting with ffc88bf4e9bff60094fae209d765b911a9d9897cbc9fd3909113faac1d9ab8d6 not found: ID does not exist"
Sep 16 17:40:06.547141 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:06.547124    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-btrjg"]
Sep 16 17:40:07.065027 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.064995    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 16 17:40:07.145543 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.145519    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:40:07.192086 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.192061    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"]
Sep 16 17:40:07.192537 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.192524    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kserve-container"
Sep 16 17:40:07.192590 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.192541    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kserve-container"
Sep 16 17:40:07.192590 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.192568    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kube-rbac-proxy"
Sep 16 17:40:07.192590 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.192576    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kube-rbac-proxy"
Sep 16 17:40:07.192590 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.192586    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="storage-initializer"
Sep 16 17:40:07.192709 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.192595    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="storage-initializer"
Sep 16 17:40:07.192709 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.192666    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kube-rbac-proxy"
Sep 16 17:40:07.192709 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.192678    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" containerName="kserve-container"
Sep 16 17:40:07.197475 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.197457    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.200463 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.200436    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 16 17:40:07.200861 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.200842    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 16 17:40:07.207037 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.207013    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"]
Sep 16 17:40:07.324775 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.324720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-9vkgd\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.324775 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.324773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-9vkgd\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.324925 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.324817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d8337f8-865f-4ea4-b17f-ed53c2d66126-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-9vkgd\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.324925 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.324832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97ggn\" (UniqueName: \"kubernetes.io/projected/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kube-api-access-97ggn\") pod \"isvc-predictive-xgboost-predictor-695f978978-9vkgd\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.324925 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.324847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d8337f8-865f-4ea4-b17f-ed53c2d66126-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-9vkgd\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.369004 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.368975    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:40:07.430908 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.430644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-9vkgd\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.430908 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.430731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-9vkgd\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.430908 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.430782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d8337f8-865f-4ea4-b17f-ed53c2d66126-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-9vkgd\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.430908 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.430806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97ggn\" (UniqueName: \"kubernetes.io/projected/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kube-api-access-97ggn\") pod \"isvc-predictive-xgboost-predictor-695f978978-9vkgd\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.430908 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.430832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d8337f8-865f-4ea4-b17f-ed53c2d66126-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-9vkgd\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.431517 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.431461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-9vkgd\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.431641 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.431620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d8337f8-865f-4ea4-b17f-ed53c2d66126-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-9vkgd\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.431992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.431761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-9vkgd\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.433940 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.433885    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d8337f8-865f-4ea4-b17f-ed53c2d66126-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-9vkgd\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.441587 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.441566    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97ggn\" (UniqueName: \"kubernetes.io/projected/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kube-api-access-97ggn\") pod \"isvc-predictive-xgboost-predictor-695f978978-9vkgd\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.507838 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.507816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:07.767410 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.767347    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b121d39d-642c-41ef-be51-f261c23a9ad8" path="/var/lib/kubelet/pods/b121d39d-642c-41ef-be51-f261c23a9ad8/volumes"
Sep 16 17:40:07.783326 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:07.783305    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"]
Sep 16 17:40:07.785048 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:40:07.785024    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8337f8_865f_4ea4_b17f_ed53c2d66126.slice/crio-3877afa7a6ae4d03d2dc79566e89e70263343d16d2b292966d29c0df2fcf479f WatchSource:0}: Error finding container 3877afa7a6ae4d03d2dc79566e89e70263343d16d2b292966d29c0df2fcf479f: Status 404 returned error can't find the container with id 3877afa7a6ae4d03d2dc79566e89e70263343d16d2b292966d29c0df2fcf479f
Sep 16 17:40:08.464634 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:08.464597    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:5000: connect: connection refused"
Sep 16 17:40:08.465050 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:08.465026    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:40:08.534993 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:08.534969    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" event={"ID":"7d8337f8-865f-4ea4-b17f-ed53c2d66126","Type":"ContainerStarted","Data":"ed9bbdc4fd03bd4a68eafb8e6254f2ef6d0323b6ae9efc3043f99754cd5dbc55"}
Sep 16 17:40:08.535080 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:08.535023    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" event={"ID":"7d8337f8-865f-4ea4-b17f-ed53c2d66126","Type":"ContainerStarted","Data":"3877afa7a6ae4d03d2dc79566e89e70263343d16d2b292966d29c0df2fcf479f"}
Sep 16 17:40:09.081488 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:09.081456    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 17:40:09.081921 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:09.081568    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:40:09.083021 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:09.082995    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:40:09.083146 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:09.083092    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:40:12.065196 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:12.065167    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 16 17:40:12.560505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:12.560472    2581 generic.go:356] "Generic (PLEG): container finished" podID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerID="ed9bbdc4fd03bd4a68eafb8e6254f2ef6d0323b6ae9efc3043f99754cd5dbc55" exitCode=0
Sep 16 17:40:12.560635 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:12.560525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" event={"ID":"7d8337f8-865f-4ea4-b17f-ed53c2d66126","Type":"ContainerDied","Data":"ed9bbdc4fd03bd4a68eafb8e6254f2ef6d0323b6ae9efc3043f99754cd5dbc55"}
Sep 16 17:40:13.568319 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:13.568285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" event={"ID":"7d8337f8-865f-4ea4-b17f-ed53c2d66126","Type":"ContainerStarted","Data":"ea3fde906e28daf3e5713307fa96e096258b73501ce81aa03439c0738ec44187"}
Sep 16 17:40:13.570715 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:13.568351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" event={"ID":"7d8337f8-865f-4ea4-b17f-ed53c2d66126","Type":"ContainerStarted","Data":"253673d2b7ed4c7ae3ede2e35309a6634acc041a0b8d5124dfb1ea19ddeb7ac9"}
Sep 16 17:40:13.570715 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:13.568543    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:13.595106 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:13.595068    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" podStartSLOduration=2.873634763 podStartE2EDuration="6.595055657s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.721420894s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:40:13.593783414 +0000 UTC m=+942.382405945" watchObservedRunningTime="2026-09-16 17:40:13.595055657 +0000 UTC m=+942.383678188"
Sep 16 17:40:13.758523 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:13.758503    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:40:13.820718 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:13.820665    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:40:14.573313 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:14.573280    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:14.574622 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:14.574588    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 17:40:15.577357 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:15.577319    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 17:40:17.065283 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.065247    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 16 17:40:17.145077 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.145053    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:40:17.305261 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.305237    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-cabundle-cert\") pod \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") "
Sep 16 17:40:17.305379 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.305288    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrg7j\" (UniqueName: \"kubernetes.io/projected/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kube-api-access-lrg7j\") pod \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") "
Sep 16 17:40:17.305379 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.305311    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kserve-provision-location\") pod \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") "
Sep 16 17:40:17.305379 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.305333    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kube-rbac-proxy-sar-config\") pod \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") "
Sep 16 17:40:17.305379 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.305357    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-proxy-tls\") pod \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\" (UID: \"8b9ae155-6e6d-46d9-bf38-d350dde1c93f\") "
Sep 16 17:40:17.305682 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.305657    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b9ae155-6e6d-46d9-bf38-d350dde1c93f" (UID: "8b9ae155-6e6d-46d9-bf38-d350dde1c93f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:40:17.305748 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.305676    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b9ae155-6e6d-46d9-bf38-d350dde1c93f" (UID: "8b9ae155-6e6d-46d9-bf38-d350dde1c93f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:40:17.305748 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.305683    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b9ae155-6e6d-46d9-bf38-d350dde1c93f" (UID: "8b9ae155-6e6d-46d9-bf38-d350dde1c93f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:40:17.307375 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.307336    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kube-api-access-lrg7j" (OuterVolumeSpecName: "kube-api-access-lrg7j") pod "8b9ae155-6e6d-46d9-bf38-d350dde1c93f" (UID: "8b9ae155-6e6d-46d9-bf38-d350dde1c93f"). InnerVolumeSpecName "kube-api-access-lrg7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:40:17.307589 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.307570    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b9ae155-6e6d-46d9-bf38-d350dde1c93f" (UID: "8b9ae155-6e6d-46d9-bf38-d350dde1c93f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:40:17.406081 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.406056    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrg7j\" (UniqueName: \"kubernetes.io/projected/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kube-api-access-lrg7j\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:40:17.406081 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.406076    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:40:17.406197 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.406086    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:40:17.406197 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.406096    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:40:17.406197 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.406105    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b9ae155-6e6d-46d9-bf38-d350dde1c93f-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:40:17.588802 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.588727    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerID="cee606cd85f6d93de65f0f20fd1f28306504257db0d13e2ea26c8cdea1a92f19" exitCode=0
Sep 16 17:40:17.588802 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.588793    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"
Sep 16 17:40:17.588998 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.588814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" event={"ID":"8b9ae155-6e6d-46d9-bf38-d350dde1c93f","Type":"ContainerDied","Data":"cee606cd85f6d93de65f0f20fd1f28306504257db0d13e2ea26c8cdea1a92f19"}
Sep 16 17:40:17.588998 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.588879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs" event={"ID":"8b9ae155-6e6d-46d9-bf38-d350dde1c93f","Type":"ContainerDied","Data":"8efceef30e26a83073d0e66a00547a5d7cb0272304fa71051738f4ea1473e1cd"}
Sep 16 17:40:17.588998 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.588928    2581 scope.go:117] "RemoveContainer" containerID="cee606cd85f6d93de65f0f20fd1f28306504257db0d13e2ea26c8cdea1a92f19"
Sep 16 17:40:17.597544 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.597522    2581 scope.go:117] "RemoveContainer" containerID="8af4349fb4e602a238d6210d3b15e6dd9dca0d400a6a7a5ab3c8805c079106d7"
Sep 16 17:40:17.604968 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.604948    2581 scope.go:117] "RemoveContainer" containerID="2d51581073d960fd3d1fcc7f3b4e333053ca1f19753c08d3a01c71783acc58a9"
Sep 16 17:40:17.611858 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.611838    2581 scope.go:117] "RemoveContainer" containerID="8b3b80eebe895c447c3323c2e1f0ee868f766eb60055809d1968149593f6d857"
Sep 16 17:40:17.616221 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.616201    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"]
Sep 16 17:40:17.620056 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.620040    2581 scope.go:117] "RemoveContainer" containerID="cee606cd85f6d93de65f0f20fd1f28306504257db0d13e2ea26c8cdea1a92f19"
Sep 16 17:40:17.623841 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.623795    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-4h7qs"]
Sep 16 17:40:17.626288 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:40:17.626262    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee606cd85f6d93de65f0f20fd1f28306504257db0d13e2ea26c8cdea1a92f19\": container with ID starting with cee606cd85f6d93de65f0f20fd1f28306504257db0d13e2ea26c8cdea1a92f19 not found: ID does not exist" containerID="cee606cd85f6d93de65f0f20fd1f28306504257db0d13e2ea26c8cdea1a92f19"
Sep 16 17:40:17.626374 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.626292    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee606cd85f6d93de65f0f20fd1f28306504257db0d13e2ea26c8cdea1a92f19"} err="failed to get container status \"cee606cd85f6d93de65f0f20fd1f28306504257db0d13e2ea26c8cdea1a92f19\": rpc error: code = NotFound desc = could not find container \"cee606cd85f6d93de65f0f20fd1f28306504257db0d13e2ea26c8cdea1a92f19\": container with ID starting with cee606cd85f6d93de65f0f20fd1f28306504257db0d13e2ea26c8cdea1a92f19 not found: ID does not exist"
Sep 16 17:40:17.626462 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.626307    2581 scope.go:117] "RemoveContainer" containerID="8af4349fb4e602a238d6210d3b15e6dd9dca0d400a6a7a5ab3c8805c079106d7"
Sep 16 17:40:17.626713 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:40:17.626696    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af4349fb4e602a238d6210d3b15e6dd9dca0d400a6a7a5ab3c8805c079106d7\": container with ID starting with 8af4349fb4e602a238d6210d3b15e6dd9dca0d400a6a7a5ab3c8805c079106d7 not found: ID does not exist" containerID="8af4349fb4e602a238d6210d3b15e6dd9dca0d400a6a7a5ab3c8805c079106d7"
Sep 16 17:40:17.626759 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.626717    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af4349fb4e602a238d6210d3b15e6dd9dca0d400a6a7a5ab3c8805c079106d7"} err="failed to get container status \"8af4349fb4e602a238d6210d3b15e6dd9dca0d400a6a7a5ab3c8805c079106d7\": rpc error: code = NotFound desc = could not find container \"8af4349fb4e602a238d6210d3b15e6dd9dca0d400a6a7a5ab3c8805c079106d7\": container with ID starting with 8af4349fb4e602a238d6210d3b15e6dd9dca0d400a6a7a5ab3c8805c079106d7 not found: ID does not exist"
Sep 16 17:40:17.626759 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.626729    2581 scope.go:117] "RemoveContainer" containerID="2d51581073d960fd3d1fcc7f3b4e333053ca1f19753c08d3a01c71783acc58a9"
Sep 16 17:40:17.626988 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:40:17.626964    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d51581073d960fd3d1fcc7f3b4e333053ca1f19753c08d3a01c71783acc58a9\": container with ID starting with 2d51581073d960fd3d1fcc7f3b4e333053ca1f19753c08d3a01c71783acc58a9 not found: ID does not exist" containerID="2d51581073d960fd3d1fcc7f3b4e333053ca1f19753c08d3a01c71783acc58a9"
Sep 16 17:40:17.627038 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.626998    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d51581073d960fd3d1fcc7f3b4e333053ca1f19753c08d3a01c71783acc58a9"} err="failed to get container status \"2d51581073d960fd3d1fcc7f3b4e333053ca1f19753c08d3a01c71783acc58a9\": rpc error: code = NotFound desc = could not find container \"2d51581073d960fd3d1fcc7f3b4e333053ca1f19753c08d3a01c71783acc58a9\": container with ID starting with 2d51581073d960fd3d1fcc7f3b4e333053ca1f19753c08d3a01c71783acc58a9 not found: ID does not exist"
Sep 16 17:40:17.627038 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.627017    2581 scope.go:117] "RemoveContainer" containerID="8b3b80eebe895c447c3323c2e1f0ee868f766eb60055809d1968149593f6d857"
Sep 16 17:40:17.627267 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:40:17.627248    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b3b80eebe895c447c3323c2e1f0ee868f766eb60055809d1968149593f6d857\": container with ID starting with 8b3b80eebe895c447c3323c2e1f0ee868f766eb60055809d1968149593f6d857 not found: ID does not exist" containerID="8b3b80eebe895c447c3323c2e1f0ee868f766eb60055809d1968149593f6d857"
Sep 16 17:40:17.627309 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.627272    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3b80eebe895c447c3323c2e1f0ee868f766eb60055809d1968149593f6d857"} err="failed to get container status \"8b3b80eebe895c447c3323c2e1f0ee868f766eb60055809d1968149593f6d857\": rpc error: code = NotFound desc = could not find container \"8b3b80eebe895c447c3323c2e1f0ee868f766eb60055809d1968149593f6d857\": container with ID starting with 8b3b80eebe895c447c3323c2e1f0ee868f766eb60055809d1968149593f6d857 not found: ID does not exist"
Sep 16 17:40:17.771214 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:17.771191    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" path="/var/lib/kubelet/pods/8b9ae155-6e6d-46d9-bf38-d350dde1c93f/volumes"
Sep 16 17:40:18.464927 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:18.464886    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:5000: connect: connection refused"
Sep 16 17:40:18.465552 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:18.465529    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:40:20.581800 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:20.581771    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:40:20.582259 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:20.582232    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 17:40:28.465372 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:28.465320    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:5000: connect: connection refused"
Sep 16 17:40:28.465952 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:28.465810    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:40:30.582542 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:30.582506    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 17:40:38.464514 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:38.464466    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:5000: connect: connection refused"
Sep 16 17:40:38.465020 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:38.464994    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:40:40.582605 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:40.582571    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 17:40:48.464991 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:48.464945    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:5000: connect: connection refused"
Sep 16 17:40:48.465449 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:48.465363    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:40:50.582433 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:50.582397    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 17:40:58.464987 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:58.464909    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:40:58.465378 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:40:58.465033    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:41:00.582238 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:00.582204    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 17:41:10.582323 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:10.582280    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 17:41:11.954470 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:11.954437    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"]
Sep 16 17:41:11.954952 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:11.954693    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kserve-container" containerID="cri-o://0814f158b27db7b25bad18757b0e1f27458a433024e562810650a41d9dd31bd9" gracePeriod=30
Sep 16 17:41:11.954952 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:11.954765    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="agent" containerID="cri-o://e1cb4dd8eea3710998f077954e5a5ae779b1c8bb1d96720f6bff305d9821d247" gracePeriod=30
Sep 16 17:41:11.954952 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:11.954859    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kube-rbac-proxy" containerID="cri-o://f88d5b61d38ac2332ff474fc37baa926e727d84cd91fe83af80499135d5dc8d1" gracePeriod=30
Sep 16 17:41:11.958877 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:11.958855    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:41:12.047315 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:12.047285    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:41:12.220083 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:12.220011    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:41:12.824785 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:12.824755    2581 generic.go:356] "Generic (PLEG): container finished" podID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerID="f88d5b61d38ac2332ff474fc37baa926e727d84cd91fe83af80499135d5dc8d1" exitCode=2
Sep 16 17:41:12.824955 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:12.824829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" event={"ID":"edb9decc-819c-4c22-94b6-0b0e37d1efb2","Type":"ContainerDied","Data":"f88d5b61d38ac2332ff474fc37baa926e727d84cd91fe83af80499135d5dc8d1"}
Sep 16 17:41:13.461118 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:13.461072    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 16 17:41:15.838777 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:15.838750    2581 generic.go:356] "Generic (PLEG): container finished" podID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerID="0814f158b27db7b25bad18757b0e1f27458a433024e562810650a41d9dd31bd9" exitCode=0
Sep 16 17:41:15.839083 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:15.838784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" event={"ID":"edb9decc-819c-4c22-94b6-0b0e37d1efb2","Type":"ContainerDied","Data":"0814f158b27db7b25bad18757b0e1f27458a433024e562810650a41d9dd31bd9"}
Sep 16 17:41:18.460976 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:18.460928    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 16 17:41:18.465285 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:18.465255    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:5000: connect: connection refused"
Sep 16 17:41:18.465567 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:18.465537    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:41:20.583668 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:20.583645    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:41:23.460267 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:23.460233    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 16 17:41:23.460608 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:23.460355    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:41:27.486229 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:27.486198    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"]
Sep 16 17:41:27.486582 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:27.486428    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kserve-container" containerID="cri-o://253673d2b7ed4c7ae3ede2e35309a6634acc041a0b8d5124dfb1ea19ddeb7ac9" gracePeriod=30
Sep 16 17:41:27.486582 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:27.486443    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kube-rbac-proxy" containerID="cri-o://ea3fde906e28daf3e5713307fa96e096258b73501ce81aa03439c0738ec44187" gracePeriod=30
Sep 16 17:41:27.494171 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:27.494147    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:41:27.887367 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:27.887344    2581 generic.go:356] "Generic (PLEG): container finished" podID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerID="ea3fde906e28daf3e5713307fa96e096258b73501ce81aa03439c0738ec44187" exitCode=2
Sep 16 17:41:27.887485 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:27.887414    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" event={"ID":"7d8337f8-865f-4ea4-b17f-ed53c2d66126","Type":"ContainerDied","Data":"ea3fde906e28daf3e5713307fa96e096258b73501ce81aa03439c0738ec44187"}
Sep 16 17:41:28.460681 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:28.460644    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 16 17:41:28.465041 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:28.465016    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:5000: connect: connection refused"
Sep 16 17:41:28.465369 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:28.465350    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:41:30.577661 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:30.577620    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 16 17:41:30.582985 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:30.582959    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 17:41:30.901649 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:30.901624    2581 generic.go:356] "Generic (PLEG): container finished" podID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerID="253673d2b7ed4c7ae3ede2e35309a6634acc041a0b8d5124dfb1ea19ddeb7ac9" exitCode=0
Sep 16 17:41:30.901765 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:30.901696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" event={"ID":"7d8337f8-865f-4ea4-b17f-ed53c2d66126","Type":"ContainerDied","Data":"253673d2b7ed4c7ae3ede2e35309a6634acc041a0b8d5124dfb1ea19ddeb7ac9"}
Sep 16 17:41:30.927347 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:30.927326    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:41:31.028028 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.028007    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d8337f8-865f-4ea4-b17f-ed53c2d66126-cabundle-cert\") pod \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") "
Sep 16 17:41:31.028145 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.028041    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kserve-provision-location\") pod \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") "
Sep 16 17:41:31.028145 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.028062    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kube-rbac-proxy-sar-config\") pod \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") "
Sep 16 17:41:31.028145 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.028079    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d8337f8-865f-4ea4-b17f-ed53c2d66126-proxy-tls\") pod \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") "
Sep 16 17:41:31.028145 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.028094    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97ggn\" (UniqueName: \"kubernetes.io/projected/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kube-api-access-97ggn\") pod \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\" (UID: \"7d8337f8-865f-4ea4-b17f-ed53c2d66126\") "
Sep 16 17:41:31.028427 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.028376    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d8337f8-865f-4ea4-b17f-ed53c2d66126" (UID: "7d8337f8-865f-4ea4-b17f-ed53c2d66126"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:41:31.028427 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.028412    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7d8337f8-865f-4ea4-b17f-ed53c2d66126" (UID: "7d8337f8-865f-4ea4-b17f-ed53c2d66126"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:41:31.028506 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.028427    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d8337f8-865f-4ea4-b17f-ed53c2d66126-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d8337f8-865f-4ea4-b17f-ed53c2d66126" (UID: "7d8337f8-865f-4ea4-b17f-ed53c2d66126"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:41:31.030210 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.030185    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d8337f8-865f-4ea4-b17f-ed53c2d66126-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7d8337f8-865f-4ea4-b17f-ed53c2d66126" (UID: "7d8337f8-865f-4ea4-b17f-ed53c2d66126"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:41:31.030274 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.030230    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kube-api-access-97ggn" (OuterVolumeSpecName: "kube-api-access-97ggn") pod "7d8337f8-865f-4ea4-b17f-ed53c2d66126" (UID: "7d8337f8-865f-4ea4-b17f-ed53c2d66126"). InnerVolumeSpecName "kube-api-access-97ggn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:41:31.128629 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.128586    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:41:31.128629 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.128605    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d8337f8-865f-4ea4-b17f-ed53c2d66126-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:41:31.128629 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.128617    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97ggn\" (UniqueName: \"kubernetes.io/projected/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kube-api-access-97ggn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:41:31.128629 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.128625    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d8337f8-865f-4ea4-b17f-ed53c2d66126-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:41:31.128791 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.128633    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d8337f8-865f-4ea4-b17f-ed53c2d66126-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:41:31.906815 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.906795    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"
Sep 16 17:41:31.907130 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.906796    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd" event={"ID":"7d8337f8-865f-4ea4-b17f-ed53c2d66126","Type":"ContainerDied","Data":"3877afa7a6ae4d03d2dc79566e89e70263343d16d2b292966d29c0df2fcf479f"}
Sep 16 17:41:31.907130 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.906947    2581 scope.go:117] "RemoveContainer" containerID="ea3fde906e28daf3e5713307fa96e096258b73501ce81aa03439c0738ec44187"
Sep 16 17:41:31.914770 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.914754    2581 scope.go:117] "RemoveContainer" containerID="253673d2b7ed4c7ae3ede2e35309a6634acc041a0b8d5124dfb1ea19ddeb7ac9"
Sep 16 17:41:31.921643 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.921629    2581 scope.go:117] "RemoveContainer" containerID="ed9bbdc4fd03bd4a68eafb8e6254f2ef6d0323b6ae9efc3043f99754cd5dbc55"
Sep 16 17:41:31.930930 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.930909    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"]
Sep 16 17:41:31.934581 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:31.934562    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9vkgd"]
Sep 16 17:41:32.086613 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.086560    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:41:32.135509 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.135482    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"]
Sep 16 17:41:32.135813 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.135799    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container"
Sep 16 17:41:32.135860 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.135815    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container"
Sep 16 17:41:32.135860 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.135830    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="storage-initializer"
Sep 16 17:41:32.135860 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.135835    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="storage-initializer"
Sep 16 17:41:32.135860 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.135844    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent"
Sep 16 17:41:32.135860 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.135849    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent"
Sep 16 17:41:32.135860 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.135858    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="storage-initializer"
Sep 16 17:41:32.135860 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.135863    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="storage-initializer"
Sep 16 17:41:32.136100 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.135872    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kube-rbac-proxy"
Sep 16 17:41:32.136100 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.135877    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kube-rbac-proxy"
Sep 16 17:41:32.136100 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.135884    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kube-rbac-proxy"
Sep 16 17:41:32.136100 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.136031    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kube-rbac-proxy"
Sep 16 17:41:32.136100 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.136042    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kserve-container"
Sep 16 17:41:32.136100 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.136047    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kserve-container"
Sep 16 17:41:32.136264 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.136106    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kube-rbac-proxy"
Sep 16 17:41:32.136264 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.136116    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="agent"
Sep 16 17:41:32.136264 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.136122    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kube-rbac-proxy"
Sep 16 17:41:32.136264 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.136127    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9ae155-6e6d-46d9-bf38-d350dde1c93f" containerName="kserve-container"
Sep 16 17:41:32.136264 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.136135    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" containerName="kserve-container"
Sep 16 17:41:32.139213 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.139198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.142063 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.142028    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 16 17:41:32.142226 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.142209    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 16 17:41:32.152232 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.152206    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"]
Sep 16 17:41:32.164070 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.164053    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:41:32.236957 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.236933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz4sp\" (UniqueName: \"kubernetes.io/projected/b30c96c9-e804-4647-a13f-0cc95f703c61-kube-api-access-pz4sp\") pod \"isvc-logger-predictor-85cd966f49-xnvgz\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.237044 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.236959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b30c96c9-e804-4647-a13f-0cc95f703c61-cabundle-cert\") pod \"isvc-logger-predictor-85cd966f49-xnvgz\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.237044 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.236985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b30c96c9-e804-4647-a13f-0cc95f703c61-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-85cd966f49-xnvgz\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.237044 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.237009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b30c96c9-e804-4647-a13f-0cc95f703c61-proxy-tls\") pod \"isvc-logger-predictor-85cd966f49-xnvgz\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.237162 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.237051    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b30c96c9-e804-4647-a13f-0cc95f703c61-kserve-provision-location\") pod \"isvc-logger-predictor-85cd966f49-xnvgz\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.337351 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.337302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz4sp\" (UniqueName: \"kubernetes.io/projected/b30c96c9-e804-4647-a13f-0cc95f703c61-kube-api-access-pz4sp\") pod \"isvc-logger-predictor-85cd966f49-xnvgz\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.337351 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.337330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b30c96c9-e804-4647-a13f-0cc95f703c61-cabundle-cert\") pod \"isvc-logger-predictor-85cd966f49-xnvgz\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.337495 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.337357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b30c96c9-e804-4647-a13f-0cc95f703c61-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-85cd966f49-xnvgz\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.337592 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.337565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b30c96c9-e804-4647-a13f-0cc95f703c61-proxy-tls\") pod \"isvc-logger-predictor-85cd966f49-xnvgz\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.337692 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.337609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b30c96c9-e804-4647-a13f-0cc95f703c61-kserve-provision-location\") pod \"isvc-logger-predictor-85cd966f49-xnvgz\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.337999 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.337979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b30c96c9-e804-4647-a13f-0cc95f703c61-cabundle-cert\") pod \"isvc-logger-predictor-85cd966f49-xnvgz\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.338080 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.337996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b30c96c9-e804-4647-a13f-0cc95f703c61-kserve-provision-location\") pod \"isvc-logger-predictor-85cd966f49-xnvgz\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.338122 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.338074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b30c96c9-e804-4647-a13f-0cc95f703c61-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-85cd966f49-xnvgz\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.339880 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.339862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b30c96c9-e804-4647-a13f-0cc95f703c61-proxy-tls\") pod \"isvc-logger-predictor-85cd966f49-xnvgz\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.346317 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.346296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz4sp\" (UniqueName: \"kubernetes.io/projected/b30c96c9-e804-4647-a13f-0cc95f703c61-kube-api-access-pz4sp\") pod \"isvc-logger-predictor-85cd966f49-xnvgz\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.447988 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.447964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:32.659373 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.659297    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:41:32.703362 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.703338    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"]
Sep 16 17:41:32.707902 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.707873    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.710760 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.710742    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 16 17:41:32.710865 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.710777    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 16 17:41:32.717959 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.717937    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"]
Sep 16 17:41:32.723701 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.723563    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"]
Sep 16 17:41:32.726253 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:41:32.726230    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30c96c9_e804_4647_a13f_0cc95f703c61.slice/crio-835b781282c1b0509227e9179561277ac2dc31ae0028e3c1587c0e2c79eaf0de WatchSource:0}: Error finding container 835b781282c1b0509227e9179561277ac2dc31ae0028e3c1587c0e2c79eaf0de: Status 404 returned error can't find the container with id 835b781282c1b0509227e9179561277ac2dc31ae0028e3c1587c0e2c79eaf0de
Sep 16 17:41:32.739820 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.739802    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f43331de-7018-42e8-8fba-bb6f6208871e-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.739864 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.739836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc8ds\" (UniqueName: \"kubernetes.io/projected/f43331de-7018-42e8-8fba-bb6f6208871e-kube-api-access-rc8ds\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.739922 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.739871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f43331de-7018-42e8-8fba-bb6f6208871e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.739922 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.739917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f43331de-7018-42e8-8fba-bb6f6208871e-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.739998 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.739970    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f43331de-7018-42e8-8fba-bb6f6208871e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.820345 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.820322    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:41:32.840902 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.840870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f43331de-7018-42e8-8fba-bb6f6208871e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.840994 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.840962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f43331de-7018-42e8-8fba-bb6f6208871e-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.841058 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.841015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc8ds\" (UniqueName: \"kubernetes.io/projected/f43331de-7018-42e8-8fba-bb6f6208871e-kube-api-access-rc8ds\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.841113 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.841078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f43331de-7018-42e8-8fba-bb6f6208871e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.841169 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.841108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f43331de-7018-42e8-8fba-bb6f6208871e-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.841613 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.841576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f43331de-7018-42e8-8fba-bb6f6208871e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.841613 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.841576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f43331de-7018-42e8-8fba-bb6f6208871e-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.841729 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.841584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f43331de-7018-42e8-8fba-bb6f6208871e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.843404 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.843386    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f43331de-7018-42e8-8fba-bb6f6208871e-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.853502 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.853484    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc8ds\" (UniqueName: \"kubernetes.io/projected/f43331de-7018-42e8-8fba-bb6f6208871e-kube-api-access-rc8ds\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:32.914442 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.914383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" event={"ID":"b30c96c9-e804-4647-a13f-0cc95f703c61","Type":"ContainerStarted","Data":"75aba70239d04ef74c37f8f51d5a3ad688202031de5c96818983dea5af4f8c48"}
Sep 16 17:41:32.914743 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:32.914446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" event={"ID":"b30c96c9-e804-4647-a13f-0cc95f703c61","Type":"ContainerStarted","Data":"835b781282c1b0509227e9179561277ac2dc31ae0028e3c1587c0e2c79eaf0de"}
Sep 16 17:41:33.018795 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:33.018773    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:33.295790 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:33.293732    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"]
Sep 16 17:41:33.298902 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:41:33.298860    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43331de_7018_42e8_8fba_bb6f6208871e.slice/crio-2b9d96a524f0a6356b631e5ea79f030874f057b12c75398875dac478b139a531 WatchSource:0}: Error finding container 2b9d96a524f0a6356b631e5ea79f030874f057b12c75398875dac478b139a531: Status 404 returned error can't find the container with id 2b9d96a524f0a6356b631e5ea79f030874f057b12c75398875dac478b139a531
Sep 16 17:41:33.460856 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:33.460821    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 16 17:41:33.767362 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:33.767283    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d8337f8-865f-4ea4-b17f-ed53c2d66126" path="/var/lib/kubelet/pods/7d8337f8-865f-4ea4-b17f-ed53c2d66126/volumes"
Sep 16 17:41:33.921038 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:33.921011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" event={"ID":"f43331de-7018-42e8-8fba-bb6f6208871e","Type":"ContainerStarted","Data":"8e444f3b971ba6e246ca1f7d11e68e91f282f13430aad3ea23145edbbf650a78"}
Sep 16 17:41:33.921459 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:33.921063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" event={"ID":"f43331de-7018-42e8-8fba-bb6f6208871e","Type":"ContainerStarted","Data":"2b9d96a524f0a6356b631e5ea79f030874f057b12c75398875dac478b139a531"}
Sep 16 17:41:36.935881 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:36.935852    2581 generic.go:356] "Generic (PLEG): container finished" podID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerID="75aba70239d04ef74c37f8f51d5a3ad688202031de5c96818983dea5af4f8c48" exitCode=0
Sep 16 17:41:36.936202 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:36.935927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" event={"ID":"b30c96c9-e804-4647-a13f-0cc95f703c61","Type":"ContainerDied","Data":"75aba70239d04ef74c37f8f51d5a3ad688202031de5c96818983dea5af4f8c48"}
Sep 16 17:41:37.945787 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:37.945753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" event={"ID":"b30c96c9-e804-4647-a13f-0cc95f703c61","Type":"ContainerStarted","Data":"be7cb25b204bc193c08a52ab87f12590293c9e8a80fe0591d0e501bf9d600896"}
Sep 16 17:41:37.946209 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:37.945823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" event={"ID":"b30c96c9-e804-4647-a13f-0cc95f703c61","Type":"ContainerStarted","Data":"d51b32409537de3788cd14f6aac890f7988d6c9e336ea2bf3862d0bb25d398ea"}
Sep 16 17:41:37.946209 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:37.945855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" event={"ID":"b30c96c9-e804-4647-a13f-0cc95f703c61","Type":"ContainerStarted","Data":"b9cdf5cfdcea3d6876002a73caf87e333d979c27931e2fa14edaba6e10a31aa5"}
Sep 16 17:41:37.946209 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:37.946184    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:37.946375 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:37.946349    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:37.947315 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:37.947296    2581 generic.go:356] "Generic (PLEG): container finished" podID="f43331de-7018-42e8-8fba-bb6f6208871e" containerID="8e444f3b971ba6e246ca1f7d11e68e91f282f13430aad3ea23145edbbf650a78" exitCode=0
Sep 16 17:41:37.947423 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:37.947362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" event={"ID":"f43331de-7018-42e8-8fba-bb6f6208871e","Type":"ContainerDied","Data":"8e444f3b971ba6e246ca1f7d11e68e91f282f13430aad3ea23145edbbf650a78"}
Sep 16 17:41:37.947738 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:37.947715    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 17:41:38.007238 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:38.007191    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podStartSLOduration=2.03913341 podStartE2EDuration="6.007174016s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.968040606s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:41:37.980807016 +0000 UTC m=+1026.769429547" watchObservedRunningTime="2026-09-16 17:41:38.007174016 +0000 UTC m=+1026.795796549"
Sep 16 17:41:38.460351 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:38.460306    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 16 17:41:38.464741 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:38.464710    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:5000: connect: connection refused"
Sep 16 17:41:38.464942 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:38.464927    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:41:38.465163 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:38.465142    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:41:38.465256 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:38.465242    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:41:38.954206 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:38.954181    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" event={"ID":"f43331de-7018-42e8-8fba-bb6f6208871e","Type":"ContainerStarted","Data":"dce354d0dd4433fdaeef3c638945fef651b7c6ef8faafc843ba24b1625903e73"}
Sep 16 17:41:38.954542 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:38.954225    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" event={"ID":"f43331de-7018-42e8-8fba-bb6f6208871e","Type":"ContainerStarted","Data":"e6b90ecbe3aeb2f43df9b5a5b446b57ed51432ff892afb6fc33e97d7d5d18aba"}
Sep 16 17:41:38.954617 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:38.954599    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:38.954753 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:38.954725    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 17:41:38.955813 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:38.955786    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:41:38.981762 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:38.981717    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" podStartSLOduration=3.221881799 podStartE2EDuration="6.981704339s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.75982254s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:41:38.980755921 +0000 UTC m=+1027.769378452" watchObservedRunningTime="2026-09-16 17:41:38.981704339 +0000 UTC m=+1027.770326931"
Sep 16 17:41:39.959146 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:39.959104    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 17:41:39.959727 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:39.959696    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:41:42.109947 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.109926    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:41:42.215067 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.215002    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kube-rbac-proxy-sar-config\") pod \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") "
Sep 16 17:41:42.215067 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.215041    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcqfw\" (UniqueName: \"kubernetes.io/projected/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kube-api-access-vcqfw\") pod \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") "
Sep 16 17:41:42.215272 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.215105    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb9decc-819c-4c22-94b6-0b0e37d1efb2-proxy-tls\") pod \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") "
Sep 16 17:41:42.215272 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.215161    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kserve-provision-location\") pod \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") "
Sep 16 17:41:42.215272 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.215189    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edb9decc-819c-4c22-94b6-0b0e37d1efb2-cabundle-cert\") pod \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\" (UID: \"edb9decc-819c-4c22-94b6-0b0e37d1efb2\") "
Sep 16 17:41:42.215457 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.215343    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "edb9decc-819c-4c22-94b6-0b0e37d1efb2" (UID: "edb9decc-819c-4c22-94b6-0b0e37d1efb2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:41:42.215515 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.215474    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "edb9decc-819c-4c22-94b6-0b0e37d1efb2" (UID: "edb9decc-819c-4c22-94b6-0b0e37d1efb2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:41:42.215572 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.215548    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb9decc-819c-4c22-94b6-0b0e37d1efb2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "edb9decc-819c-4c22-94b6-0b0e37d1efb2" (UID: "edb9decc-819c-4c22-94b6-0b0e37d1efb2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:41:42.217201 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.217178    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kube-api-access-vcqfw" (OuterVolumeSpecName: "kube-api-access-vcqfw") pod "edb9decc-819c-4c22-94b6-0b0e37d1efb2" (UID: "edb9decc-819c-4c22-94b6-0b0e37d1efb2"). InnerVolumeSpecName "kube-api-access-vcqfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:41:42.217267 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.217248    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edb9decc-819c-4c22-94b6-0b0e37d1efb2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "edb9decc-819c-4c22-94b6-0b0e37d1efb2" (UID: "edb9decc-819c-4c22-94b6-0b0e37d1efb2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:41:42.315629 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.315608    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:41:42.315629 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.315628    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcqfw\" (UniqueName: \"kubernetes.io/projected/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kube-api-access-vcqfw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:41:42.315759 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.315637    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edb9decc-819c-4c22-94b6-0b0e37d1efb2-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:41:42.315759 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.315647    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edb9decc-819c-4c22-94b6-0b0e37d1efb2-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:41:42.315759 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.315655    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edb9decc-819c-4c22-94b6-0b0e37d1efb2-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:41:42.977142 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.977113    2581 generic.go:356] "Generic (PLEG): container finished" podID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerID="e1cb4dd8eea3710998f077954e5a5ae779b1c8bb1d96720f6bff305d9821d247" exitCode=0
Sep 16 17:41:42.977260 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.977180    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"
Sep 16 17:41:42.977260 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.977189    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" event={"ID":"edb9decc-819c-4c22-94b6-0b0e37d1efb2","Type":"ContainerDied","Data":"e1cb4dd8eea3710998f077954e5a5ae779b1c8bb1d96720f6bff305d9821d247"}
Sep 16 17:41:42.977260 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.977245    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m" event={"ID":"edb9decc-819c-4c22-94b6-0b0e37d1efb2","Type":"ContainerDied","Data":"2280d4c0d7317c953ab8939b51459c00fae2fb2288c5e2bbb219e802418b6e50"}
Sep 16 17:41:42.977355 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.977274    2581 scope.go:117] "RemoveContainer" containerID="e1cb4dd8eea3710998f077954e5a5ae779b1c8bb1d96720f6bff305d9821d247"
Sep 16 17:41:42.985383 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.985369    2581 scope.go:117] "RemoveContainer" containerID="f88d5b61d38ac2332ff474fc37baa926e727d84cd91fe83af80499135d5dc8d1"
Sep 16 17:41:42.992797 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.992781    2581 scope.go:117] "RemoveContainer" containerID="0814f158b27db7b25bad18757b0e1f27458a433024e562810650a41d9dd31bd9"
Sep 16 17:41:42.999689 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:42.999672    2581 scope.go:117] "RemoveContainer" containerID="1e90afaa6a6ab706def46b93813d42b3c4875c3e45c625074d7a0462f278a2b1"
Sep 16 17:41:43.006614 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.006594    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"]
Sep 16 17:41:43.006828 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.006813    2581 scope.go:117] "RemoveContainer" containerID="e1cb4dd8eea3710998f077954e5a5ae779b1c8bb1d96720f6bff305d9821d247"
Sep 16 17:41:43.007138 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:41:43.007118    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1cb4dd8eea3710998f077954e5a5ae779b1c8bb1d96720f6bff305d9821d247\": container with ID starting with e1cb4dd8eea3710998f077954e5a5ae779b1c8bb1d96720f6bff305d9821d247 not found: ID does not exist" containerID="e1cb4dd8eea3710998f077954e5a5ae779b1c8bb1d96720f6bff305d9821d247"
Sep 16 17:41:43.007215 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.007147    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1cb4dd8eea3710998f077954e5a5ae779b1c8bb1d96720f6bff305d9821d247"} err="failed to get container status \"e1cb4dd8eea3710998f077954e5a5ae779b1c8bb1d96720f6bff305d9821d247\": rpc error: code = NotFound desc = could not find container \"e1cb4dd8eea3710998f077954e5a5ae779b1c8bb1d96720f6bff305d9821d247\": container with ID starting with e1cb4dd8eea3710998f077954e5a5ae779b1c8bb1d96720f6bff305d9821d247 not found: ID does not exist"
Sep 16 17:41:43.007215 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.007168    2581 scope.go:117] "RemoveContainer" containerID="f88d5b61d38ac2332ff474fc37baa926e727d84cd91fe83af80499135d5dc8d1"
Sep 16 17:41:43.007441 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:41:43.007416    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88d5b61d38ac2332ff474fc37baa926e727d84cd91fe83af80499135d5dc8d1\": container with ID starting with f88d5b61d38ac2332ff474fc37baa926e727d84cd91fe83af80499135d5dc8d1 not found: ID does not exist" containerID="f88d5b61d38ac2332ff474fc37baa926e727d84cd91fe83af80499135d5dc8d1"
Sep 16 17:41:43.007558 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.007450    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88d5b61d38ac2332ff474fc37baa926e727d84cd91fe83af80499135d5dc8d1"} err="failed to get container status \"f88d5b61d38ac2332ff474fc37baa926e727d84cd91fe83af80499135d5dc8d1\": rpc error: code = NotFound desc = could not find container \"f88d5b61d38ac2332ff474fc37baa926e727d84cd91fe83af80499135d5dc8d1\": container with ID starting with f88d5b61d38ac2332ff474fc37baa926e727d84cd91fe83af80499135d5dc8d1 not found: ID does not exist"
Sep 16 17:41:43.007558 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.007479    2581 scope.go:117] "RemoveContainer" containerID="0814f158b27db7b25bad18757b0e1f27458a433024e562810650a41d9dd31bd9"
Sep 16 17:41:43.007960 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:41:43.007935    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0814f158b27db7b25bad18757b0e1f27458a433024e562810650a41d9dd31bd9\": container with ID starting with 0814f158b27db7b25bad18757b0e1f27458a433024e562810650a41d9dd31bd9 not found: ID does not exist" containerID="0814f158b27db7b25bad18757b0e1f27458a433024e562810650a41d9dd31bd9"
Sep 16 17:41:43.008058 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.007964    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0814f158b27db7b25bad18757b0e1f27458a433024e562810650a41d9dd31bd9"} err="failed to get container status \"0814f158b27db7b25bad18757b0e1f27458a433024e562810650a41d9dd31bd9\": rpc error: code = NotFound desc = could not find container \"0814f158b27db7b25bad18757b0e1f27458a433024e562810650a41d9dd31bd9\": container with ID starting with 0814f158b27db7b25bad18757b0e1f27458a433024e562810650a41d9dd31bd9 not found: ID does not exist"
Sep 16 17:41:43.008058 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.007983    2581 scope.go:117] "RemoveContainer" containerID="1e90afaa6a6ab706def46b93813d42b3c4875c3e45c625074d7a0462f278a2b1"
Sep 16 17:41:43.008343 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:41:43.008316    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e90afaa6a6ab706def46b93813d42b3c4875c3e45c625074d7a0462f278a2b1\": container with ID starting with 1e90afaa6a6ab706def46b93813d42b3c4875c3e45c625074d7a0462f278a2b1 not found: ID does not exist" containerID="1e90afaa6a6ab706def46b93813d42b3c4875c3e45c625074d7a0462f278a2b1"
Sep 16 17:41:43.008435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.008347    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e90afaa6a6ab706def46b93813d42b3c4875c3e45c625074d7a0462f278a2b1"} err="failed to get container status \"1e90afaa6a6ab706def46b93813d42b3c4875c3e45c625074d7a0462f278a2b1\": rpc error: code = NotFound desc = could not find container \"1e90afaa6a6ab706def46b93813d42b3c4875c3e45c625074d7a0462f278a2b1\": container with ID starting with 1e90afaa6a6ab706def46b93813d42b3c4875c3e45c625074d7a0462f278a2b1 not found: ID does not exist"
Sep 16 17:41:43.009664 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.009647    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-9wp9m"]
Sep 16 17:41:43.766314 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.766278    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" path="/var/lib/kubelet/pods/edb9decc-819c-4c22-94b6-0b0e37d1efb2/volumes"
Sep 16 17:41:43.954404 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.954378    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:43.954778 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.954753    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:43.955944 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.955917    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 17:41:43.959435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.959413    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:41:43.982569 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:43.982547    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 17:41:44.963537 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:44.963512    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:41:44.964095 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:44.964056    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 17:41:44.964485 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:44.964457    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:41:44.986941 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:44.986918    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 17:41:54.964046 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:54.963981    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 17:41:54.964503 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:54.964477    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:41:54.987575 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:41:54.987548    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 17:42:04.964050 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:04.963991    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 17:42:04.964638 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:04.964442    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:42:04.987636 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:04.987607    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 17:42:05.199783 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:05.199742    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:42:14.964732 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:14.964693    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 17:42:14.965243 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:14.965224    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:42:14.986964 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:14.986930    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 17:42:24.964547 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:24.964327    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 17:42:24.964999 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:24.964801    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:42:24.987098 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:24.987069    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 17:42:34.964011 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:34.963966    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 17:42:34.964524 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:34.964501    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:42:34.987800 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:34.987778    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 17:42:44.964713 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:44.964674    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:42:44.965146 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:44.964769    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:42:44.986952 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:44.986917    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 17:42:54.987949 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:54.987885    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:42:57.324942 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.324883    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"]
Sep 16 17:42:57.325400 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.325179    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container" containerID="cri-o://b9cdf5cfdcea3d6876002a73caf87e333d979c27931e2fa14edaba6e10a31aa5" gracePeriod=30
Sep 16 17:42:57.325400 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.325220    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="agent" containerID="cri-o://be7cb25b204bc193c08a52ab87f12590293c9e8a80fe0591d0e501bf9d600896" gracePeriod=30
Sep 16 17:42:57.325400 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.325209    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kube-rbac-proxy" containerID="cri-o://d51b32409537de3788cd14f6aac890f7988d6c9e336ea2bf3862d0bb25d398ea" gracePeriod=30
Sep 16 17:42:57.332738 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.332713    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:42:57.457026 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.456999    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"]
Sep 16 17:42:57.457361 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.457348    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="storage-initializer"
Sep 16 17:42:57.457424 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.457366    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="storage-initializer"
Sep 16 17:42:57.457424 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.457378    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kserve-container"
Sep 16 17:42:57.457424 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.457384    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kserve-container"
Sep 16 17:42:57.457424 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.457390    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kube-rbac-proxy"
Sep 16 17:42:57.457424 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.457395    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kube-rbac-proxy"
Sep 16 17:42:57.457424 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.457407    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="agent"
Sep 16 17:42:57.457424 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.457412    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="agent"
Sep 16 17:42:57.457774 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.457463    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kube-rbac-proxy"
Sep 16 17:42:57.457774 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.457471    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="kserve-container"
Sep 16 17:42:57.457774 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.457477    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb9decc-819c-4c22-94b6-0b0e37d1efb2" containerName="agent"
Sep 16 17:42:57.460653 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.460637    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.463383 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.463365    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 16 17:42:57.463468 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.463373    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 16 17:42:57.472444 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.472423    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"]
Sep 16 17:42:57.601137 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.601059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj77q\" (UniqueName: \"kubernetes.io/projected/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kube-api-access-vj77q\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dzb9x\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.601137 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.601111    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dzb9x\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.601303 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.601139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dzb9x\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.601303 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.601165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dzb9x\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.601303 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.601197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dzb9x\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.702445 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.702418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dzb9x\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.702577 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.702490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj77q\" (UniqueName: \"kubernetes.io/projected/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kube-api-access-vj77q\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dzb9x\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.702577 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.702549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dzb9x\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.702696 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.702574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dzb9x\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.702696 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.702602    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dzb9x\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.702829 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.702804    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dzb9x\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.703244 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.703227    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dzb9x\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.703301 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.703253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dzb9x\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.705112 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.705087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dzb9x\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.713970 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.713946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj77q\" (UniqueName: \"kubernetes.io/projected/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kube-api-access-vj77q\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dzb9x\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:57.771732 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:57.771716    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:42:58.043519 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:58.043499    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"]
Sep 16 17:42:58.045529 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:42:58.045504    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49783d2_3b08_4ac9_bc47_ccf600c2a8c5.slice/crio-37a68d1454a41d8097a4238fb6c02b754d78c7f7e64b0740bc5f4647bc77f580 WatchSource:0}: Error finding container 37a68d1454a41d8097a4238fb6c02b754d78c7f7e64b0740bc5f4647bc77f580: Status 404 returned error can't find the container with id 37a68d1454a41d8097a4238fb6c02b754d78c7f7e64b0740bc5f4647bc77f580
Sep 16 17:42:58.310391 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:58.310319    2581 generic.go:356] "Generic (PLEG): container finished" podID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerID="d51b32409537de3788cd14f6aac890f7988d6c9e336ea2bf3862d0bb25d398ea" exitCode=2
Sep 16 17:42:58.310545 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:58.310402    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" event={"ID":"b30c96c9-e804-4647-a13f-0cc95f703c61","Type":"ContainerDied","Data":"d51b32409537de3788cd14f6aac890f7988d6c9e336ea2bf3862d0bb25d398ea"}
Sep 16 17:42:58.311992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:58.311966    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x" event={"ID":"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5","Type":"ContainerStarted","Data":"23ebbaa8b77eb8da8857c7bc5b7735a4a21866af235549ee071994cb832e9db6"}
Sep 16 17:42:58.312089 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:58.312014    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x" event={"ID":"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5","Type":"ContainerStarted","Data":"37a68d1454a41d8097a4238fb6c02b754d78c7f7e64b0740bc5f4647bc77f580"}
Sep 16 17:42:58.756217 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:58.756096    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:42:59.959768 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:42:59.959706    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 16 17:43:01.328707 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:01.328645    2581 generic.go:356] "Generic (PLEG): container finished" podID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerID="b9cdf5cfdcea3d6876002a73caf87e333d979c27931e2fa14edaba6e10a31aa5" exitCode=0
Sep 16 17:43:01.329019 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:01.328711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" event={"ID":"b30c96c9-e804-4647-a13f-0cc95f703c61","Type":"ContainerDied","Data":"b9cdf5cfdcea3d6876002a73caf87e333d979c27931e2fa14edaba6e10a31aa5"}
Sep 16 17:43:02.931346 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:02.931315    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"]
Sep 16 17:43:02.931692 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:02.931538    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kserve-container" containerID="cri-o://e6b90ecbe3aeb2f43df9b5a5b446b57ed51432ff892afb6fc33e97d7d5d18aba" gracePeriod=30
Sep 16 17:43:02.931692 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:02.931613    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kube-rbac-proxy" containerID="cri-o://dce354d0dd4433fdaeef3c638945fef651b7c6ef8faafc843ba24b1625903e73" gracePeriod=30
Sep 16 17:43:02.949449 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:02.949426    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:43:03.339239 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:03.339214    2581 generic.go:356] "Generic (PLEG): container finished" podID="f43331de-7018-42e8-8fba-bb6f6208871e" containerID="dce354d0dd4433fdaeef3c638945fef651b7c6ef8faafc843ba24b1625903e73" exitCode=2
Sep 16 17:43:03.339342 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:03.339288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" event={"ID":"f43331de-7018-42e8-8fba-bb6f6208871e","Type":"ContainerDied","Data":"dce354d0dd4433fdaeef3c638945fef651b7c6ef8faafc843ba24b1625903e73"}
Sep 16 17:43:03.960250 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:03.960217    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 16 17:43:04.959701 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:04.959658    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 16 17:43:04.963930 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:04.963879    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 17:43:04.964266 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:04.964244    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:43:04.987032 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:04.987007    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 17:43:07.170045 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.170020    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:43:07.273773 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.273752    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc8ds\" (UniqueName: \"kubernetes.io/projected/f43331de-7018-42e8-8fba-bb6f6208871e-kube-api-access-rc8ds\") pod \"f43331de-7018-42e8-8fba-bb6f6208871e\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") "
Sep 16 17:43:07.273924 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.273784    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f43331de-7018-42e8-8fba-bb6f6208871e-kserve-provision-location\") pod \"f43331de-7018-42e8-8fba-bb6f6208871e\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") "
Sep 16 17:43:07.273924 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.273812    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f43331de-7018-42e8-8fba-bb6f6208871e-proxy-tls\") pod \"f43331de-7018-42e8-8fba-bb6f6208871e\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") "
Sep 16 17:43:07.273924 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.273859    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f43331de-7018-42e8-8fba-bb6f6208871e-kube-rbac-proxy-sar-config\") pod \"f43331de-7018-42e8-8fba-bb6f6208871e\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") "
Sep 16 17:43:07.274098 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.273956    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f43331de-7018-42e8-8fba-bb6f6208871e-cabundle-cert\") pod \"f43331de-7018-42e8-8fba-bb6f6208871e\" (UID: \"f43331de-7018-42e8-8fba-bb6f6208871e\") "
Sep 16 17:43:07.274205 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.274169    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43331de-7018-42e8-8fba-bb6f6208871e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f43331de-7018-42e8-8fba-bb6f6208871e" (UID: "f43331de-7018-42e8-8fba-bb6f6208871e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:07.274353 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.274325    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43331de-7018-42e8-8fba-bb6f6208871e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f43331de-7018-42e8-8fba-bb6f6208871e" (UID: "f43331de-7018-42e8-8fba-bb6f6208871e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:43:07.274393 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.274330    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43331de-7018-42e8-8fba-bb6f6208871e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f43331de-7018-42e8-8fba-bb6f6208871e" (UID: "f43331de-7018-42e8-8fba-bb6f6208871e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:43:07.276029 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.276003    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43331de-7018-42e8-8fba-bb6f6208871e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f43331de-7018-42e8-8fba-bb6f6208871e" (UID: "f43331de-7018-42e8-8fba-bb6f6208871e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:43:07.276029 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.276014    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43331de-7018-42e8-8fba-bb6f6208871e-kube-api-access-rc8ds" (OuterVolumeSpecName: "kube-api-access-rc8ds") pod "f43331de-7018-42e8-8fba-bb6f6208871e" (UID: "f43331de-7018-42e8-8fba-bb6f6208871e"). InnerVolumeSpecName "kube-api-access-rc8ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:43:07.358700 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.358675    2581 generic.go:356] "Generic (PLEG): container finished" podID="f43331de-7018-42e8-8fba-bb6f6208871e" containerID="e6b90ecbe3aeb2f43df9b5a5b446b57ed51432ff892afb6fc33e97d7d5d18aba" exitCode=0
Sep 16 17:43:07.358786 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.358727    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"
Sep 16 17:43:07.358786 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.358761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" event={"ID":"f43331de-7018-42e8-8fba-bb6f6208871e","Type":"ContainerDied","Data":"e6b90ecbe3aeb2f43df9b5a5b446b57ed51432ff892afb6fc33e97d7d5d18aba"}
Sep 16 17:43:07.358856 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.358817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7" event={"ID":"f43331de-7018-42e8-8fba-bb6f6208871e","Type":"ContainerDied","Data":"2b9d96a524f0a6356b631e5ea79f030874f057b12c75398875dac478b139a531"}
Sep 16 17:43:07.358856 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.358846    2581 scope.go:117] "RemoveContainer" containerID="dce354d0dd4433fdaeef3c638945fef651b7c6ef8faafc843ba24b1625903e73"
Sep 16 17:43:07.367342 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.367322    2581 scope.go:117] "RemoveContainer" containerID="e6b90ecbe3aeb2f43df9b5a5b446b57ed51432ff892afb6fc33e97d7d5d18aba"
Sep 16 17:43:07.375715 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.375580    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f43331de-7018-42e8-8fba-bb6f6208871e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:43:07.375715 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.375605    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f43331de-7018-42e8-8fba-bb6f6208871e-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:43:07.375715 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.375620    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc8ds\" (UniqueName: \"kubernetes.io/projected/f43331de-7018-42e8-8fba-bb6f6208871e-kube-api-access-rc8ds\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:43:07.375715 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.375631    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f43331de-7018-42e8-8fba-bb6f6208871e-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:43:07.375715 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.375643    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f43331de-7018-42e8-8fba-bb6f6208871e-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:43:07.376466 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.376451    2581 scope.go:117] "RemoveContainer" containerID="8e444f3b971ba6e246ca1f7d11e68e91f282f13430aad3ea23145edbbf650a78"
Sep 16 17:43:07.382006 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.381987    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"]
Sep 16 17:43:07.384088 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.384073    2581 scope.go:117] "RemoveContainer" containerID="dce354d0dd4433fdaeef3c638945fef651b7c6ef8faafc843ba24b1625903e73"
Sep 16 17:43:07.384337 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:43:07.384323    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce354d0dd4433fdaeef3c638945fef651b7c6ef8faafc843ba24b1625903e73\": container with ID starting with dce354d0dd4433fdaeef3c638945fef651b7c6ef8faafc843ba24b1625903e73 not found: ID does not exist" containerID="dce354d0dd4433fdaeef3c638945fef651b7c6ef8faafc843ba24b1625903e73"
Sep 16 17:43:07.384371 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.384349    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce354d0dd4433fdaeef3c638945fef651b7c6ef8faafc843ba24b1625903e73"} err="failed to get container status \"dce354d0dd4433fdaeef3c638945fef651b7c6ef8faafc843ba24b1625903e73\": rpc error: code = NotFound desc = could not find container \"dce354d0dd4433fdaeef3c638945fef651b7c6ef8faafc843ba24b1625903e73\": container with ID starting with dce354d0dd4433fdaeef3c638945fef651b7c6ef8faafc843ba24b1625903e73 not found: ID does not exist"
Sep 16 17:43:07.384371 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.384364    2581 scope.go:117] "RemoveContainer" containerID="e6b90ecbe3aeb2f43df9b5a5b446b57ed51432ff892afb6fc33e97d7d5d18aba"
Sep 16 17:43:07.384611 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:43:07.384593    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6b90ecbe3aeb2f43df9b5a5b446b57ed51432ff892afb6fc33e97d7d5d18aba\": container with ID starting with e6b90ecbe3aeb2f43df9b5a5b446b57ed51432ff892afb6fc33e97d7d5d18aba not found: ID does not exist" containerID="e6b90ecbe3aeb2f43df9b5a5b446b57ed51432ff892afb6fc33e97d7d5d18aba"
Sep 16 17:43:07.384643 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.384626    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6b90ecbe3aeb2f43df9b5a5b446b57ed51432ff892afb6fc33e97d7d5d18aba"} err="failed to get container status \"e6b90ecbe3aeb2f43df9b5a5b446b57ed51432ff892afb6fc33e97d7d5d18aba\": rpc error: code = NotFound desc = could not find container \"e6b90ecbe3aeb2f43df9b5a5b446b57ed51432ff892afb6fc33e97d7d5d18aba\": container with ID starting with e6b90ecbe3aeb2f43df9b5a5b446b57ed51432ff892afb6fc33e97d7d5d18aba not found: ID does not exist"
Sep 16 17:43:07.384643 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.384641    2581 scope.go:117] "RemoveContainer" containerID="8e444f3b971ba6e246ca1f7d11e68e91f282f13430aad3ea23145edbbf650a78"
Sep 16 17:43:07.384850 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:43:07.384832    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e444f3b971ba6e246ca1f7d11e68e91f282f13430aad3ea23145edbbf650a78\": container with ID starting with 8e444f3b971ba6e246ca1f7d11e68e91f282f13430aad3ea23145edbbf650a78 not found: ID does not exist" containerID="8e444f3b971ba6e246ca1f7d11e68e91f282f13430aad3ea23145edbbf650a78"
Sep 16 17:43:07.384921 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.384850    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e444f3b971ba6e246ca1f7d11e68e91f282f13430aad3ea23145edbbf650a78"} err="failed to get container status \"8e444f3b971ba6e246ca1f7d11e68e91f282f13430aad3ea23145edbbf650a78\": rpc error: code = NotFound desc = could not find container \"8e444f3b971ba6e246ca1f7d11e68e91f282f13430aad3ea23145edbbf650a78\": container with ID starting with 8e444f3b971ba6e246ca1f7d11e68e91f282f13430aad3ea23145edbbf650a78 not found: ID does not exist"
Sep 16 17:43:07.388625 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.388605    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-7fkh7"]
Sep 16 17:43:07.766076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:07.766011    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" path="/var/lib/kubelet/pods/f43331de-7018-42e8-8fba-bb6f6208871e/volumes"
Sep 16 17:43:08.092666 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:08.092635    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:43:08.287064 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:08.287037    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:43:09.960149 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:09.960105    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 16 17:43:09.960575 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:09.960235    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:43:14.960206 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:14.960167    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 16 17:43:14.964554 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:14.964525    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 17:43:14.964825 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:14.964804    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:43:19.959516 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:19.959453    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 16 17:43:20.416167 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:20.416141    2581 generic.go:356] "Generic (PLEG): container finished" podID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerID="23ebbaa8b77eb8da8857c7bc5b7735a4a21866af235549ee071994cb832e9db6" exitCode=0
Sep 16 17:43:20.416300 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:20.416222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x" event={"ID":"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5","Type":"ContainerDied","Data":"23ebbaa8b77eb8da8857c7bc5b7735a4a21866af235549ee071994cb832e9db6"}
Sep 16 17:43:20.417453 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:20.417438    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:43:24.960336 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:24.960292    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 16 17:43:24.964661 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:24.964625    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 17:43:24.964781 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:24.964766    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:43:24.965222 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:24.965190    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:43:24.965334 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:24.965316    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:43:27.459169 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.459138    2581 generic.go:356] "Generic (PLEG): container finished" podID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerID="be7cb25b204bc193c08a52ab87f12590293c9e8a80fe0591d0e501bf9d600896" exitCode=0
Sep 16 17:43:27.459548 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.459167    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" event={"ID":"b30c96c9-e804-4647-a13f-0cc95f703c61","Type":"ContainerDied","Data":"be7cb25b204bc193c08a52ab87f12590293c9e8a80fe0591d0e501bf9d600896"}
Sep 16 17:43:27.511267 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.511248    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:43:27.626996 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.626920    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b30c96c9-e804-4647-a13f-0cc95f703c61-kube-rbac-proxy-sar-config\") pod \"b30c96c9-e804-4647-a13f-0cc95f703c61\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") "
Sep 16 17:43:27.626996 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.626963    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b30c96c9-e804-4647-a13f-0cc95f703c61-proxy-tls\") pod \"b30c96c9-e804-4647-a13f-0cc95f703c61\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") "
Sep 16 17:43:27.627188 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.627029    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz4sp\" (UniqueName: \"kubernetes.io/projected/b30c96c9-e804-4647-a13f-0cc95f703c61-kube-api-access-pz4sp\") pod \"b30c96c9-e804-4647-a13f-0cc95f703c61\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") "
Sep 16 17:43:27.627188 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.627071    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b30c96c9-e804-4647-a13f-0cc95f703c61-cabundle-cert\") pod \"b30c96c9-e804-4647-a13f-0cc95f703c61\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") "
Sep 16 17:43:27.627188 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.627114    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b30c96c9-e804-4647-a13f-0cc95f703c61-kserve-provision-location\") pod \"b30c96c9-e804-4647-a13f-0cc95f703c61\" (UID: \"b30c96c9-e804-4647-a13f-0cc95f703c61\") "
Sep 16 17:43:27.627349 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.627324    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30c96c9-e804-4647-a13f-0cc95f703c61-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b30c96c9-e804-4647-a13f-0cc95f703c61" (UID: "b30c96c9-e804-4647-a13f-0cc95f703c61"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:43:27.627527 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.627508    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b30c96c9-e804-4647-a13f-0cc95f703c61-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:43:27.627612 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.627587    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30c96c9-e804-4647-a13f-0cc95f703c61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b30c96c9-e804-4647-a13f-0cc95f703c61" (UID: "b30c96c9-e804-4647-a13f-0cc95f703c61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:27.627671 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.627613    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30c96c9-e804-4647-a13f-0cc95f703c61-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b30c96c9-e804-4647-a13f-0cc95f703c61" (UID: "b30c96c9-e804-4647-a13f-0cc95f703c61"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:43:27.629700 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.629631    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30c96c9-e804-4647-a13f-0cc95f703c61-kube-api-access-pz4sp" (OuterVolumeSpecName: "kube-api-access-pz4sp") pod "b30c96c9-e804-4647-a13f-0cc95f703c61" (UID: "b30c96c9-e804-4647-a13f-0cc95f703c61"). InnerVolumeSpecName "kube-api-access-pz4sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:43:27.629810 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.629711    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b30c96c9-e804-4647-a13f-0cc95f703c61-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b30c96c9-e804-4647-a13f-0cc95f703c61" (UID: "b30c96c9-e804-4647-a13f-0cc95f703c61"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:43:27.728728 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.728706    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b30c96c9-e804-4647-a13f-0cc95f703c61-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:43:27.728846 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.728730    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b30c96c9-e804-4647-a13f-0cc95f703c61-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:43:27.728846 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.728745    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz4sp\" (UniqueName: \"kubernetes.io/projected/b30c96c9-e804-4647-a13f-0cc95f703c61-kube-api-access-pz4sp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:43:27.728846 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:27.728756    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b30c96c9-e804-4647-a13f-0cc95f703c61-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:43:28.466487 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:28.466454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz" event={"ID":"b30c96c9-e804-4647-a13f-0cc95f703c61","Type":"ContainerDied","Data":"835b781282c1b0509227e9179561277ac2dc31ae0028e3c1587c0e2c79eaf0de"}
Sep 16 17:43:28.466487 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:28.466484    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"
Sep 16 17:43:28.467012 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:28.466524    2581 scope.go:117] "RemoveContainer" containerID="be7cb25b204bc193c08a52ab87f12590293c9e8a80fe0591d0e501bf9d600896"
Sep 16 17:43:28.474449 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:28.474433    2581 scope.go:117] "RemoveContainer" containerID="d51b32409537de3788cd14f6aac890f7988d6c9e336ea2bf3862d0bb25d398ea"
Sep 16 17:43:28.481652 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:28.481634    2581 scope.go:117] "RemoveContainer" containerID="b9cdf5cfdcea3d6876002a73caf87e333d979c27931e2fa14edaba6e10a31aa5"
Sep 16 17:43:28.489083 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:28.489060    2581 scope.go:117] "RemoveContainer" containerID="75aba70239d04ef74c37f8f51d5a3ad688202031de5c96818983dea5af4f8c48"
Sep 16 17:43:28.493435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:28.493380    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"]
Sep 16 17:43:28.496245 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:28.496226    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-xnvgz"]
Sep 16 17:43:29.767871 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:29.767827    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" path="/var/lib/kubelet/pods/b30c96c9-e804-4647-a13f-0cc95f703c61/volumes"
Sep 16 17:43:48.172626 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:43:48.172591    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:31.743450 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:44:31.743419    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:44:31.744437 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:44:31.744359    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:44:58.268237 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:44:58.268181    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:03.357208 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:03.357171    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:03.527139 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:03.527110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:16.826854 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:16.826791    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:24.025853 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:24.025815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x" event={"ID":"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5","Type":"ContainerStarted","Data":"d29a975bcdae8b9a906bd5815fac00d40ede980fdc9d196667e168fcd3c2e718"}
Sep 16 17:45:24.026357 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:24.025911    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x" event={"ID":"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5","Type":"ContainerStarted","Data":"798accd170e647bedc1212b4d9f80977a5fe06fe9ec5d1db21725f9670524e4c"}
Sep 16 17:45:24.026357 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:24.025954    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:45:24.026357 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:24.025988    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:45:24.027326 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:24.027283    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x" podUID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 17:45:24.050088 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:24.050038    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x" podStartSLOduration=2.734157856 podStartE2EDuration="2m27.050022377s" totalImagesPullingTime="2m2.770600104s" totalInitContainerRuntime="21.545264417s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:45:24.048869774 +0000 UTC m=+1252.837492505" watchObservedRunningTime="2026-09-16 17:45:24.050022377 +0000 UTC m=+1252.838644909"
Sep 16 17:45:25.036583 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:25.030957    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x" podUID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 17:45:30.035909 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:30.035872    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:45:30.036574 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:30.036550    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x" podUID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 17:45:40.037573 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:40.037544    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:45:53.621978 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:53.621939    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"]
Sep 16 17:45:53.622391 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:53.622237    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x" podUID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerName="kserve-container" containerID="cri-o://798accd170e647bedc1212b4d9f80977a5fe06fe9ec5d1db21725f9670524e4c" gracePeriod=30
Sep 16 17:45:53.622391 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:53.622252    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x" podUID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerName="kube-rbac-proxy" containerID="cri-o://d29a975bcdae8b9a906bd5815fac00d40ede980fdc9d196667e168fcd3c2e718" gracePeriod=30
Sep 16 17:45:53.634473 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:53.634450    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:54.156211 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:54.156173    2581 generic.go:356] "Generic (PLEG): container finished" podID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerID="d29a975bcdae8b9a906bd5815fac00d40ede980fdc9d196667e168fcd3c2e718" exitCode=2
Sep 16 17:45:54.156383 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:54.156239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x" event={"ID":"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5","Type":"ContainerDied","Data":"d29a975bcdae8b9a906bd5815fac00d40ede980fdc9d196667e168fcd3c2e718"}
Sep 16 17:45:55.090341 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.090314    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:45:55.162262 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.162170    2581 generic.go:356] "Generic (PLEG): container finished" podID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerID="798accd170e647bedc1212b4d9f80977a5fe06fe9ec5d1db21725f9670524e4c" exitCode=0
Sep 16 17:45:55.162262 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.162244    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"
Sep 16 17:45:55.162453 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.162255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x" event={"ID":"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5","Type":"ContainerDied","Data":"798accd170e647bedc1212b4d9f80977a5fe06fe9ec5d1db21725f9670524e4c"}
Sep 16 17:45:55.162453 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.162317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x" event={"ID":"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5","Type":"ContainerDied","Data":"37a68d1454a41d8097a4238fb6c02b754d78c7f7e64b0740bc5f4647bc77f580"}
Sep 16 17:45:55.162453 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.162344    2581 scope.go:117] "RemoveContainer" containerID="d29a975bcdae8b9a906bd5815fac00d40ede980fdc9d196667e168fcd3c2e718"
Sep 16 17:45:55.170436 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.170417    2581 scope.go:117] "RemoveContainer" containerID="798accd170e647bedc1212b4d9f80977a5fe06fe9ec5d1db21725f9670524e4c"
Sep 16 17:45:55.177684 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.177664    2581 scope.go:117] "RemoveContainer" containerID="23ebbaa8b77eb8da8857c7bc5b7735a4a21866af235549ee071994cb832e9db6"
Sep 16 17:45:55.184824 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.184810    2581 scope.go:117] "RemoveContainer" containerID="d29a975bcdae8b9a906bd5815fac00d40ede980fdc9d196667e168fcd3c2e718"
Sep 16 17:45:55.185092 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:45:55.185072    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d29a975bcdae8b9a906bd5815fac00d40ede980fdc9d196667e168fcd3c2e718\": container with ID starting with d29a975bcdae8b9a906bd5815fac00d40ede980fdc9d196667e168fcd3c2e718 not found: ID does not exist" containerID="d29a975bcdae8b9a906bd5815fac00d40ede980fdc9d196667e168fcd3c2e718"
Sep 16 17:45:55.185146 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.185102    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d29a975bcdae8b9a906bd5815fac00d40ede980fdc9d196667e168fcd3c2e718"} err="failed to get container status \"d29a975bcdae8b9a906bd5815fac00d40ede980fdc9d196667e168fcd3c2e718\": rpc error: code = NotFound desc = could not find container \"d29a975bcdae8b9a906bd5815fac00d40ede980fdc9d196667e168fcd3c2e718\": container with ID starting with d29a975bcdae8b9a906bd5815fac00d40ede980fdc9d196667e168fcd3c2e718 not found: ID does not exist"
Sep 16 17:45:55.185146 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.185117    2581 scope.go:117] "RemoveContainer" containerID="798accd170e647bedc1212b4d9f80977a5fe06fe9ec5d1db21725f9670524e4c"
Sep 16 17:45:55.185356 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:45:55.185339    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798accd170e647bedc1212b4d9f80977a5fe06fe9ec5d1db21725f9670524e4c\": container with ID starting with 798accd170e647bedc1212b4d9f80977a5fe06fe9ec5d1db21725f9670524e4c not found: ID does not exist" containerID="798accd170e647bedc1212b4d9f80977a5fe06fe9ec5d1db21725f9670524e4c"
Sep 16 17:45:55.185421 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.185365    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798accd170e647bedc1212b4d9f80977a5fe06fe9ec5d1db21725f9670524e4c"} err="failed to get container status \"798accd170e647bedc1212b4d9f80977a5fe06fe9ec5d1db21725f9670524e4c\": rpc error: code = NotFound desc = could not find container \"798accd170e647bedc1212b4d9f80977a5fe06fe9ec5d1db21725f9670524e4c\": container with ID starting with 798accd170e647bedc1212b4d9f80977a5fe06fe9ec5d1db21725f9670524e4c not found: ID does not exist"
Sep 16 17:45:55.185421 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.185386    2581 scope.go:117] "RemoveContainer" containerID="23ebbaa8b77eb8da8857c7bc5b7735a4a21866af235549ee071994cb832e9db6"
Sep 16 17:45:55.185639 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:45:55.185623    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ebbaa8b77eb8da8857c7bc5b7735a4a21866af235549ee071994cb832e9db6\": container with ID starting with 23ebbaa8b77eb8da8857c7bc5b7735a4a21866af235549ee071994cb832e9db6 not found: ID does not exist" containerID="23ebbaa8b77eb8da8857c7bc5b7735a4a21866af235549ee071994cb832e9db6"
Sep 16 17:45:55.185678 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.185643    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ebbaa8b77eb8da8857c7bc5b7735a4a21866af235549ee071994cb832e9db6"} err="failed to get container status \"23ebbaa8b77eb8da8857c7bc5b7735a4a21866af235549ee071994cb832e9db6\": rpc error: code = NotFound desc = could not find container \"23ebbaa8b77eb8da8857c7bc5b7735a4a21866af235549ee071994cb832e9db6\": container with ID starting with 23ebbaa8b77eb8da8857c7bc5b7735a4a21866af235549ee071994cb832e9db6 not found: ID does not exist"
Sep 16 17:45:55.248970 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.248946    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kserve-provision-location\") pod \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") "
Sep 16 17:45:55.249083 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.248974    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-cabundle-cert\") pod \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") "
Sep 16 17:45:55.249083 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.249005    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vj77q\" (UniqueName: \"kubernetes.io/projected/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kube-api-access-vj77q\") pod \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") "
Sep 16 17:45:55.249083 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.249020    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kube-rbac-proxy-sar-config\") pod \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") "
Sep 16 17:45:55.249083 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.249058    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-proxy-tls\") pod \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\" (UID: \"d49783d2-3b08-4ac9-bc47-ccf600c2a8c5\") "
Sep 16 17:45:55.249431 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.249401    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" (UID: "d49783d2-3b08-4ac9-bc47-ccf600c2a8c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:45:55.249547 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.249424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" (UID: "d49783d2-3b08-4ac9-bc47-ccf600c2a8c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:45:55.251232 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.251205    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" (UID: "d49783d2-3b08-4ac9-bc47-ccf600c2a8c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:45:55.251325 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.251247    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kube-api-access-vj77q" (OuterVolumeSpecName: "kube-api-access-vj77q") pod "d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" (UID: "d49783d2-3b08-4ac9-bc47-ccf600c2a8c5"). InnerVolumeSpecName "kube-api-access-vj77q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:45:55.255359 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.255291    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" (UID: "d49783d2-3b08-4ac9-bc47-ccf600c2a8c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:45:55.349934 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.349880    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj77q\" (UniqueName: \"kubernetes.io/projected/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kube-api-access-vj77q\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:45:55.349934 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.349931    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:45:55.349934 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.349941    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:45:55.350173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.349951    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:45:55.350173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.349969    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:45:55.488364 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.488337    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"]
Sep 16 17:45:55.498317 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.498281    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x"]
Sep 16 17:45:55.767134 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:55.767052    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" path="/var/lib/kubelet/pods/d49783d2-3b08-4ac9-bc47-ccf600c2a8c5/volumes"
Sep 16 17:45:56.031087 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:56.031051    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dzb9x" podUID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 17:45:58.807851 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:58.807816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:59.017001 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:45:59.016972    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:46:33.499965 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:33.499934    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:46:38.773765 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.773732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:46:38.852446 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852420    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"]
Sep 16 17:46:38.852725 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852714    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kserve-container"
Sep 16 17:46:38.852772 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852727    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kserve-container"
Sep 16 17:46:38.852772 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852740    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="storage-initializer"
Sep 16 17:46:38.852772 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852745    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="storage-initializer"
Sep 16 17:46:38.852772 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852754    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kube-rbac-proxy"
Sep 16 17:46:38.852772 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852759    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kube-rbac-proxy"
Sep 16 17:46:38.852772 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852765    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerName="kserve-container"
Sep 16 17:46:38.852772 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852770    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerName="kserve-container"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852777    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerName="kube-rbac-proxy"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852781    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerName="kube-rbac-proxy"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852789    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kube-rbac-proxy"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852793    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kube-rbac-proxy"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852799    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerName="storage-initializer"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852804    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerName="storage-initializer"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852810    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852815    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852823    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="agent"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852827    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="agent"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852836    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="storage-initializer"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852840    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="storage-initializer"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852885    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerName="kserve-container"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852911    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kube-rbac-proxy"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852921    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="agent"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852929    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b30c96c9-e804-4647-a13f-0cc95f703c61" containerName="kserve-container"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852938    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kserve-container"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852945    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49783d2-3b08-4ac9-bc47-ccf600c2a8c5" containerName="kube-rbac-proxy"
Sep 16 17:46:38.853173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.852953    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f43331de-7018-42e8-8fba-bb6f6208871e" containerName="kube-rbac-proxy"
Sep 16 17:46:38.857182 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.857167    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:38.861008 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.860989    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 16 17:46:38.861127 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.861044    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\""
Sep 16 17:46:38.862185 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.862167    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 17:46:38.862312 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.862182    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 17:46:38.862390 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.862194    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 17:46:38.862453 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.862253    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:46:38.862453 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.862252    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 16 17:46:38.874869 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.874851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b4136a4-26fa-4335-b126-9b0dda066b11-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:38.874970 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.874881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b4136a4-26fa-4335-b126-9b0dda066b11-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:38.874970 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.874925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b4136a4-26fa-4335-b126-9b0dda066b11-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:38.874970 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.874954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9wsk\" (UniqueName: \"kubernetes.io/projected/0b4136a4-26fa-4335-b126-9b0dda066b11-kube-api-access-j9wsk\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:38.875081 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.874984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b4136a4-26fa-4335-b126-9b0dda066b11-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:38.877509 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.877489    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"]
Sep 16 17:46:38.886209 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.886190    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:46:38.975355 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.975337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b4136a4-26fa-4335-b126-9b0dda066b11-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:38.975436 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.975367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b4136a4-26fa-4335-b126-9b0dda066b11-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:38.975436 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.975388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b4136a4-26fa-4335-b126-9b0dda066b11-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:38.975436 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.975415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9wsk\" (UniqueName: \"kubernetes.io/projected/0b4136a4-26fa-4335-b126-9b0dda066b11-kube-api-access-j9wsk\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:38.975573 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.975438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b4136a4-26fa-4335-b126-9b0dda066b11-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:38.975843 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.975824    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b4136a4-26fa-4335-b126-9b0dda066b11-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:38.976069 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.976053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b4136a4-26fa-4335-b126-9b0dda066b11-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:38.976112 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.976066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b4136a4-26fa-4335-b126-9b0dda066b11-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:38.977781 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.977754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b4136a4-26fa-4335-b126-9b0dda066b11-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:38.988438 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:38.988419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9wsk\" (UniqueName: \"kubernetes.io/projected/0b4136a4-26fa-4335-b126-9b0dda066b11-kube-api-access-j9wsk\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:39.167037 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:39.166978    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:39.447326 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:39.447304    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"]
Sep 16 17:46:39.449663 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:46:39.449633    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4136a4_26fa_4335_b126_9b0dda066b11.slice/crio-18ae3a27212878dbb3676468ad7d63cd19d1f4b9916287aec9c147bd64759e1c WatchSource:0}: Error finding container 18ae3a27212878dbb3676468ad7d63cd19d1f4b9916287aec9c147bd64759e1c: Status 404 returned error can't find the container with id 18ae3a27212878dbb3676468ad7d63cd19d1f4b9916287aec9c147bd64759e1c
Sep 16 17:46:40.343016 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:40.342986    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" event={"ID":"0b4136a4-26fa-4335-b126-9b0dda066b11","Type":"ContainerStarted","Data":"eb2f98c65ee057fc4fa122928975baa87420d4fb01dbe95cf1300767b9c82bcd"}
Sep 16 17:46:40.343359 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:40.343041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" event={"ID":"0b4136a4-26fa-4335-b126-9b0dda066b11","Type":"ContainerStarted","Data":"18ae3a27212878dbb3676468ad7d63cd19d1f4b9916287aec9c147bd64759e1c"}
Sep 16 17:46:42.651178 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:42.651149    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:46:43.356194 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:43.356168    2581 generic.go:356] "Generic (PLEG): container finished" podID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerID="eb2f98c65ee057fc4fa122928975baa87420d4fb01dbe95cf1300767b9c82bcd" exitCode=0
Sep 16 17:46:43.356291 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:43.356241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" event={"ID":"0b4136a4-26fa-4335-b126-9b0dda066b11","Type":"ContainerDied","Data":"eb2f98c65ee057fc4fa122928975baa87420d4fb01dbe95cf1300767b9c82bcd"}
Sep 16 17:46:44.362088 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:44.362057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" event={"ID":"0b4136a4-26fa-4335-b126-9b0dda066b11","Type":"ContainerStarted","Data":"78842f43111c2db317c149802e88135fb584245dd0d7447d72cb375050bd377b"}
Sep 16 17:46:44.362536 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:44.362122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" event={"ID":"0b4136a4-26fa-4335-b126-9b0dda066b11","Type":"ContainerStarted","Data":"cd5dfc347a817364da7848bd5b231a0cf002609eff25c9cb8b11b6c070e54558"}
Sep 16 17:46:44.362536 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:44.362458    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:44.362536 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:44.362501    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:46:44.384112 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:44.384062    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" podStartSLOduration=2.6018808939999998 podStartE2EDuration="6.384047779s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.782166885s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:46:44.382874026 +0000 UTC m=+1333.171496549" watchObservedRunningTime="2026-09-16 17:46:44.384047779 +0000 UTC m=+1333.172670311"
Sep 16 17:46:50.371450 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:46:50.371418    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:47:20.372566 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:47:20.372522    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 17:47:30.372237 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:47:30.372202    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 17:47:40.372043 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:47:40.372009    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 17:47:50.372877 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:47:50.372842    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 17:47:58.864740 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:47:58.864703    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:48:08.799805 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:08.799774    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"]
Sep 16 17:48:08.800247 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:08.800026    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerName="kserve-container" containerID="cri-o://cd5dfc347a817364da7848bd5b231a0cf002609eff25c9cb8b11b6c070e54558" gracePeriod=30
Sep 16 17:48:08.800247 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:08.800053    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerName="kube-rbac-proxy" containerID="cri-o://78842f43111c2db317c149802e88135fb584245dd0d7447d72cb375050bd377b" gracePeriod=30
Sep 16 17:48:08.823547 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:08.823525    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:09.710841 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:09.710809    2581 generic.go:356] "Generic (PLEG): container finished" podID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerID="78842f43111c2db317c149802e88135fb584245dd0d7447d72cb375050bd377b" exitCode=2
Sep 16 17:48:09.711013 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:09.710882    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" event={"ID":"0b4136a4-26fa-4335-b126-9b0dda066b11","Type":"ContainerDied","Data":"78842f43111c2db317c149802e88135fb584245dd0d7447d72cb375050bd377b"}
Sep 16 17:48:10.366485 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:10.366444    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 16 17:48:13.138522 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.138496    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:48:13.187776 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.187749    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b4136a4-26fa-4335-b126-9b0dda066b11-kserve-provision-location\") pod \"0b4136a4-26fa-4335-b126-9b0dda066b11\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") "
Sep 16 17:48:13.187910 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.187795    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b4136a4-26fa-4335-b126-9b0dda066b11-cabundle-cert\") pod \"0b4136a4-26fa-4335-b126-9b0dda066b11\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") "
Sep 16 17:48:13.187910 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.187837    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b4136a4-26fa-4335-b126-9b0dda066b11-proxy-tls\") pod \"0b4136a4-26fa-4335-b126-9b0dda066b11\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") "
Sep 16 17:48:13.187986 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.187921    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b4136a4-26fa-4335-b126-9b0dda066b11-kube-rbac-proxy-sar-config\") pod \"0b4136a4-26fa-4335-b126-9b0dda066b11\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") "
Sep 16 17:48:13.187986 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.187940    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9wsk\" (UniqueName: \"kubernetes.io/projected/0b4136a4-26fa-4335-b126-9b0dda066b11-kube-api-access-j9wsk\") pod \"0b4136a4-26fa-4335-b126-9b0dda066b11\" (UID: \"0b4136a4-26fa-4335-b126-9b0dda066b11\") "
Sep 16 17:48:13.188079 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.188053    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4136a4-26fa-4335-b126-9b0dda066b11-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b4136a4-26fa-4335-b126-9b0dda066b11" (UID: "0b4136a4-26fa-4335-b126-9b0dda066b11"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:48:13.188265 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.188241    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4136a4-26fa-4335-b126-9b0dda066b11-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b4136a4-26fa-4335-b126-9b0dda066b11" (UID: "0b4136a4-26fa-4335-b126-9b0dda066b11"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:48:13.188328 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.188262    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b4136a4-26fa-4335-b126-9b0dda066b11-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:48:13.188328 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.188284    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4136a4-26fa-4335-b126-9b0dda066b11-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b4136a4-26fa-4335-b126-9b0dda066b11" (UID: "0b4136a4-26fa-4335-b126-9b0dda066b11"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:48:13.190042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.190011    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b4136a4-26fa-4335-b126-9b0dda066b11-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b4136a4-26fa-4335-b126-9b0dda066b11" (UID: "0b4136a4-26fa-4335-b126-9b0dda066b11"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:48:13.190142 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.190063    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4136a4-26fa-4335-b126-9b0dda066b11-kube-api-access-j9wsk" (OuterVolumeSpecName: "kube-api-access-j9wsk") pod "0b4136a4-26fa-4335-b126-9b0dda066b11" (UID: "0b4136a4-26fa-4335-b126-9b0dda066b11"). InnerVolumeSpecName "kube-api-access-j9wsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:48:13.288681 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.288661    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b4136a4-26fa-4335-b126-9b0dda066b11-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:48:13.288763 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.288683    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b4136a4-26fa-4335-b126-9b0dda066b11-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:48:13.288763 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.288692    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b4136a4-26fa-4335-b126-9b0dda066b11-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:48:13.288763 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.288700    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9wsk\" (UniqueName: \"kubernetes.io/projected/0b4136a4-26fa-4335-b126-9b0dda066b11-kube-api-access-j9wsk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:48:13.729091 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.729021    2581 generic.go:356] "Generic (PLEG): container finished" podID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerID="cd5dfc347a817364da7848bd5b231a0cf002609eff25c9cb8b11b6c070e54558" exitCode=0
Sep 16 17:48:13.729091 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.729072    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"
Sep 16 17:48:13.729235 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.729107    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" event={"ID":"0b4136a4-26fa-4335-b126-9b0dda066b11","Type":"ContainerDied","Data":"cd5dfc347a817364da7848bd5b231a0cf002609eff25c9cb8b11b6c070e54558"}
Sep 16 17:48:13.729235 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.729193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4" event={"ID":"0b4136a4-26fa-4335-b126-9b0dda066b11","Type":"ContainerDied","Data":"18ae3a27212878dbb3676468ad7d63cd19d1f4b9916287aec9c147bd64759e1c"}
Sep 16 17:48:13.729235 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.729220    2581 scope.go:117] "RemoveContainer" containerID="78842f43111c2db317c149802e88135fb584245dd0d7447d72cb375050bd377b"
Sep 16 17:48:13.739379 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.739361    2581 scope.go:117] "RemoveContainer" containerID="cd5dfc347a817364da7848bd5b231a0cf002609eff25c9cb8b11b6c070e54558"
Sep 16 17:48:13.746997 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.746980    2581 scope.go:117] "RemoveContainer" containerID="eb2f98c65ee057fc4fa122928975baa87420d4fb01dbe95cf1300767b9c82bcd"
Sep 16 17:48:13.753925 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.753910    2581 scope.go:117] "RemoveContainer" containerID="78842f43111c2db317c149802e88135fb584245dd0d7447d72cb375050bd377b"
Sep 16 17:48:13.754160 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:48:13.754145    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78842f43111c2db317c149802e88135fb584245dd0d7447d72cb375050bd377b\": container with ID starting with 78842f43111c2db317c149802e88135fb584245dd0d7447d72cb375050bd377b not found: ID does not exist" containerID="78842f43111c2db317c149802e88135fb584245dd0d7447d72cb375050bd377b"
Sep 16 17:48:13.754221 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.754166    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78842f43111c2db317c149802e88135fb584245dd0d7447d72cb375050bd377b"} err="failed to get container status \"78842f43111c2db317c149802e88135fb584245dd0d7447d72cb375050bd377b\": rpc error: code = NotFound desc = could not find container \"78842f43111c2db317c149802e88135fb584245dd0d7447d72cb375050bd377b\": container with ID starting with 78842f43111c2db317c149802e88135fb584245dd0d7447d72cb375050bd377b not found: ID does not exist"
Sep 16 17:48:13.754221 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.754181    2581 scope.go:117] "RemoveContainer" containerID="cd5dfc347a817364da7848bd5b231a0cf002609eff25c9cb8b11b6c070e54558"
Sep 16 17:48:13.754432 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:48:13.754412    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5dfc347a817364da7848bd5b231a0cf002609eff25c9cb8b11b6c070e54558\": container with ID starting with cd5dfc347a817364da7848bd5b231a0cf002609eff25c9cb8b11b6c070e54558 not found: ID does not exist" containerID="cd5dfc347a817364da7848bd5b231a0cf002609eff25c9cb8b11b6c070e54558"
Sep 16 17:48:13.754475 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.754438    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5dfc347a817364da7848bd5b231a0cf002609eff25c9cb8b11b6c070e54558"} err="failed to get container status \"cd5dfc347a817364da7848bd5b231a0cf002609eff25c9cb8b11b6c070e54558\": rpc error: code = NotFound desc = could not find container \"cd5dfc347a817364da7848bd5b231a0cf002609eff25c9cb8b11b6c070e54558\": container with ID starting with cd5dfc347a817364da7848bd5b231a0cf002609eff25c9cb8b11b6c070e54558 not found: ID does not exist"
Sep 16 17:48:13.754475 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.754453    2581 scope.go:117] "RemoveContainer" containerID="eb2f98c65ee057fc4fa122928975baa87420d4fb01dbe95cf1300767b9c82bcd"
Sep 16 17:48:13.754692 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:48:13.754675    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2f98c65ee057fc4fa122928975baa87420d4fb01dbe95cf1300767b9c82bcd\": container with ID starting with eb2f98c65ee057fc4fa122928975baa87420d4fb01dbe95cf1300767b9c82bcd not found: ID does not exist" containerID="eb2f98c65ee057fc4fa122928975baa87420d4fb01dbe95cf1300767b9c82bcd"
Sep 16 17:48:13.754753 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.754699    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2f98c65ee057fc4fa122928975baa87420d4fb01dbe95cf1300767b9c82bcd"} err="failed to get container status \"eb2f98c65ee057fc4fa122928975baa87420d4fb01dbe95cf1300767b9c82bcd\": rpc error: code = NotFound desc = could not find container \"eb2f98c65ee057fc4fa122928975baa87420d4fb01dbe95cf1300767b9c82bcd\": container with ID starting with eb2f98c65ee057fc4fa122928975baa87420d4fb01dbe95cf1300767b9c82bcd not found: ID does not exist"
Sep 16 17:48:13.759767 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.759736    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"]
Sep 16 17:48:13.766072 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.766055    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-thhz4"]
Sep 16 17:48:13.930203 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:13.930163    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:14.123078 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:14.123045    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:15.765447 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:15.765417    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" path="/var/lib/kubelet/pods/0b4136a4-26fa-4335-b126-9b0dda066b11/volumes"
Sep 16 17:48:16.007857 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:16.007824    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:31.505108 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.505021    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-ab07eb70"]
Sep 16 17:48:31.505550 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.505370    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerName="kserve-container"
Sep 16 17:48:31.505550 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.505384    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerName="kserve-container"
Sep 16 17:48:31.505550 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.505408    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerName="storage-initializer"
Sep 16 17:48:31.505550 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.505416    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerName="storage-initializer"
Sep 16 17:48:31.505550 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.505424    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerName="kube-rbac-proxy"
Sep 16 17:48:31.505550 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.505431    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerName="kube-rbac-proxy"
Sep 16 17:48:31.505550 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.505473    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerName="kube-rbac-proxy"
Sep 16 17:48:31.505550 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.505484    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b4136a4-26fa-4335-b126-9b0dda066b11" containerName="kserve-container"
Sep 16 17:48:31.509022 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.509001    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ab07eb70"
Sep 16 17:48:31.511748 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.511729    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hlr6s\""
Sep 16 17:48:31.512068 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.512050    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 17:48:31.512915 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.512878    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 17:48:31.520014 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.519994    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-ab07eb70"]
Sep 16 17:48:31.607305 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.607283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf6qv\" (UniqueName: \"kubernetes.io/projected/f73dacd8-ab1f-4520-82f7-af9e42867dbf-kube-api-access-lf6qv\") pod \"curl-metrics-ab07eb70\" (UID: \"f73dacd8-ab1f-4520-82f7-af9e42867dbf\") " pod="kserve/curl-metrics-ab07eb70"
Sep 16 17:48:31.707570 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.707553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf6qv\" (UniqueName: \"kubernetes.io/projected/f73dacd8-ab1f-4520-82f7-af9e42867dbf-kube-api-access-lf6qv\") pod \"curl-metrics-ab07eb70\" (UID: \"f73dacd8-ab1f-4520-82f7-af9e42867dbf\") " pod="kserve/curl-metrics-ab07eb70"
Sep 16 17:48:31.718305 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.718285    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 17:48:31.729937 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.729315    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 17:48:31.737582 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.737556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf6qv\" (UniqueName: \"kubernetes.io/projected/f73dacd8-ab1f-4520-82f7-af9e42867dbf-kube-api-access-lf6qv\") pod \"curl-metrics-ab07eb70\" (UID: \"f73dacd8-ab1f-4520-82f7-af9e42867dbf\") " pod="kserve/curl-metrics-ab07eb70"
Sep 16 17:48:31.840476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.840456    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hlr6s\""
Sep 16 17:48:31.844936 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:31.844922    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ab07eb70"
Sep 16 17:48:32.137362 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:32.137279    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-ab07eb70"]
Sep 16 17:48:32.142010 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:48:32.141982    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf73dacd8_ab1f_4520_82f7_af9e42867dbf.slice/crio-2a8e96a54743f7211e69b5cf6fec071d20f169b5c5c343e409eff627c45c8dce WatchSource:0}: Error finding container 2a8e96a54743f7211e69b5cf6fec071d20f169b5c5c343e409eff627c45c8dce: Status 404 returned error can't find the container with id 2a8e96a54743f7211e69b5cf6fec071d20f169b5c5c343e409eff627c45c8dce
Sep 16 17:48:32.143574 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:32.143557    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:48:32.802329 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:32.802297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-ab07eb70" event={"ID":"f73dacd8-ab1f-4520-82f7-af9e42867dbf","Type":"ContainerStarted","Data":"2a8e96a54743f7211e69b5cf6fec071d20f169b5c5c343e409eff627c45c8dce"}
Sep 16 17:48:33.807508 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:33.807474    2581 generic.go:356] "Generic (PLEG): container finished" podID="f73dacd8-ab1f-4520-82f7-af9e42867dbf" containerID="0fdbdc6190062a1c7c42c6430a6c234d37106dbd183c732de521baa9c1a33b0a" exitCode=0
Sep 16 17:48:33.807872 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:33.807547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-ab07eb70" event={"ID":"f73dacd8-ab1f-4520-82f7-af9e42867dbf","Type":"ContainerDied","Data":"0fdbdc6190062a1c7c42c6430a6c234d37106dbd183c732de521baa9c1a33b0a"}
Sep 16 17:48:34.935411 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:34.935384    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ab07eb70"
Sep 16 17:48:34.956213 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:34.956187    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-ab07eb70_f73dacd8-ab1f-4520-82f7-af9e42867dbf/curl-metrics-ab07eb70/0.log"
Sep 16 17:48:34.983110 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:34.983087    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-ab07eb70"]
Sep 16 17:48:34.985487 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:34.985466    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-ab07eb70"]
Sep 16 17:48:35.032459 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:35.032438    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf6qv\" (UniqueName: \"kubernetes.io/projected/f73dacd8-ab1f-4520-82f7-af9e42867dbf-kube-api-access-lf6qv\") pod \"f73dacd8-ab1f-4520-82f7-af9e42867dbf\" (UID: \"f73dacd8-ab1f-4520-82f7-af9e42867dbf\") "
Sep 16 17:48:35.034415 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:35.034383    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73dacd8-ab1f-4520-82f7-af9e42867dbf-kube-api-access-lf6qv" (OuterVolumeSpecName: "kube-api-access-lf6qv") pod "f73dacd8-ab1f-4520-82f7-af9e42867dbf" (UID: "f73dacd8-ab1f-4520-82f7-af9e42867dbf"). InnerVolumeSpecName "kube-api-access-lf6qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:48:35.133145 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:35.133090    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf6qv\" (UniqueName: \"kubernetes.io/projected/f73dacd8-ab1f-4520-82f7-af9e42867dbf-kube-api-access-lf6qv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:48:35.766668 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:35.766634    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f73dacd8-ab1f-4520-82f7-af9e42867dbf" path="/var/lib/kubelet/pods/f73dacd8-ab1f-4520-82f7-af9e42867dbf/volumes"
Sep 16 17:48:35.816730 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:35.816705    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ab07eb70"
Sep 16 17:48:35.816860 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:35.816704    2581 scope.go:117] "RemoveContainer" containerID="0fdbdc6190062a1c7c42c6430a6c234d37106dbd183c732de521baa9c1a33b0a"
Sep 16 17:48:40.386130 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:40.386098    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-e6759f81"]
Sep 16 17:48:40.386549 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:40.386407    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f73dacd8-ab1f-4520-82f7-af9e42867dbf" containerName="curl-metrics-ab07eb70"
Sep 16 17:48:40.386549 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:40.386418    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f73dacd8-ab1f-4520-82f7-af9e42867dbf" containerName="curl-metrics-ab07eb70"
Sep 16 17:48:40.386549 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:40.386487    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f73dacd8-ab1f-4520-82f7-af9e42867dbf" containerName="curl-metrics-ab07eb70"
Sep 16 17:48:40.390640 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:40.390624    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-e6759f81"
Sep 16 17:48:40.393472 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:40.393452    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 17:48:40.393553 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:40.393487    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hlr6s\""
Sep 16 17:48:40.394598 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:40.394583    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 17:48:40.400603 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:40.400583    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-e6759f81"]
Sep 16 17:48:40.472467 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:40.472439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqs6f\" (UniqueName: \"kubernetes.io/projected/f5d7ad59-d4c8-49b6-abd2-325d777c81e4-kube-api-access-wqs6f\") pod \"curl-metrics-e6759f81\" (UID: \"f5d7ad59-d4c8-49b6-abd2-325d777c81e4\") " pod="kserve/curl-metrics-e6759f81"
Sep 16 17:48:40.573043 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:40.573013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqs6f\" (UniqueName: \"kubernetes.io/projected/f5d7ad59-d4c8-49b6-abd2-325d777c81e4-kube-api-access-wqs6f\") pod \"curl-metrics-e6759f81\" (UID: \"f5d7ad59-d4c8-49b6-abd2-325d777c81e4\") " pod="kserve/curl-metrics-e6759f81"
Sep 16 17:48:40.583805 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:40.583773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqs6f\" (UniqueName: \"kubernetes.io/projected/f5d7ad59-d4c8-49b6-abd2-325d777c81e4-kube-api-access-wqs6f\") pod \"curl-metrics-e6759f81\" (UID: \"f5d7ad59-d4c8-49b6-abd2-325d777c81e4\") " pod="kserve/curl-metrics-e6759f81"
Sep 16 17:48:40.716322 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:40.716245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-e6759f81"
Sep 16 17:48:40.993172 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:40.993150    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-e6759f81"]
Sep 16 17:48:40.995836 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:48:40.995791    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5d7ad59_d4c8_49b6_abd2_325d777c81e4.slice/crio-ccd70458f6aed343dee00091d16a9eca02dea94e77fdd8e83ae3862682c4ba66 WatchSource:0}: Error finding container ccd70458f6aed343dee00091d16a9eca02dea94e77fdd8e83ae3862682c4ba66: Status 404 returned error can't find the container with id ccd70458f6aed343dee00091d16a9eca02dea94e77fdd8e83ae3862682c4ba66
Sep 16 17:48:41.843113 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:41.843027    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5d7ad59-d4c8-49b6-abd2-325d777c81e4" containerID="98ac7320153b42510261a42cccc459e083f3e48e0bf1ea1f461fc0ed00d72cd0" exitCode=0
Sep 16 17:48:41.843113 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:41.843102    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-e6759f81" event={"ID":"f5d7ad59-d4c8-49b6-abd2-325d777c81e4","Type":"ContainerDied","Data":"98ac7320153b42510261a42cccc459e083f3e48e0bf1ea1f461fc0ed00d72cd0"}
Sep 16 17:48:41.843478 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:41.843138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-e6759f81" event={"ID":"f5d7ad59-d4c8-49b6-abd2-325d777c81e4","Type":"ContainerStarted","Data":"ccd70458f6aed343dee00091d16a9eca02dea94e77fdd8e83ae3862682c4ba66"}
Sep 16 17:48:42.965216 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:42.965192    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-e6759f81"
Sep 16 17:48:43.006773 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:43.006741    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-e6759f81_f5d7ad59-d4c8-49b6-abd2-325d777c81e4/curl-metrics-e6759f81/0.log"
Sep 16 17:48:43.039056 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:43.039034    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-e6759f81"]
Sep 16 17:48:43.040768 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:43.040746    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-e6759f81"]
Sep 16 17:48:43.093190 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:43.093159    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqs6f\" (UniqueName: \"kubernetes.io/projected/f5d7ad59-d4c8-49b6-abd2-325d777c81e4-kube-api-access-wqs6f\") pod \"f5d7ad59-d4c8-49b6-abd2-325d777c81e4\" (UID: \"f5d7ad59-d4c8-49b6-abd2-325d777c81e4\") "
Sep 16 17:48:43.095251 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:43.095230    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d7ad59-d4c8-49b6-abd2-325d777c81e4-kube-api-access-wqs6f" (OuterVolumeSpecName: "kube-api-access-wqs6f") pod "f5d7ad59-d4c8-49b6-abd2-325d777c81e4" (UID: "f5d7ad59-d4c8-49b6-abd2-325d777c81e4"). InnerVolumeSpecName "kube-api-access-wqs6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:48:43.194128 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:43.194086    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqs6f\" (UniqueName: \"kubernetes.io/projected/f5d7ad59-d4c8-49b6-abd2-325d777c81e4-kube-api-access-wqs6f\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:48:43.766271 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:43.766242    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5d7ad59-d4c8-49b6-abd2-325d777c81e4" path="/var/lib/kubelet/pods/f5d7ad59-d4c8-49b6-abd2-325d777c81e4/volumes"
Sep 16 17:48:43.852537 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:43.852516    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-e6759f81"
Sep 16 17:48:43.852537 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:43.852529    2581 scope.go:117] "RemoveContainer" containerID="98ac7320153b42510261a42cccc459e083f3e48e0bf1ea1f461fc0ed00d72cd0"
Sep 16 17:48:48.443463 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.443429    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:48.542016 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.541985    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"]
Sep 16 17:48:48.542291 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.542280    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d7ad59-d4c8-49b6-abd2-325d777c81e4" containerName="curl-metrics-e6759f81"
Sep 16 17:48:48.542334 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.542293    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d7ad59-d4c8-49b6-abd2-325d777c81e4" containerName="curl-metrics-e6759f81"
Sep 16 17:48:48.542371 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.542342    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d7ad59-d4c8-49b6-abd2-325d777c81e4" containerName="curl-metrics-e6759f81"
Sep 16 17:48:48.546646 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.546631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.550705 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.550660    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:48.555647 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.555627    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 17:48:48.555742 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.555640    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 17:48:48.555804 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.555754    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:48:48.556764 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.556751    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 16 17:48:48.556815 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.556796    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 17:48:48.556869 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.556827    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\""
Sep 16 17:48:48.564280 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.564262    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"]
Sep 16 17:48:48.568920 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.568876    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 16 17:48:48.634730 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.634706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41329f3e-36dd-45f3-8595-85cfb0db2a3f-cabundle-cert\") pod \"isvc-sklearn-predictor-5bd8597c94-n9d8f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.634812 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.634746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5bd8597c94-n9d8f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.634812 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.634781    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41329f3e-36dd-45f3-8595-85cfb0db2a3f-proxy-tls\") pod \"isvc-sklearn-predictor-5bd8597c94-n9d8f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.634885 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.634831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28bqz\" (UniqueName: \"kubernetes.io/projected/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kube-api-access-28bqz\") pod \"isvc-sklearn-predictor-5bd8597c94-n9d8f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.634885 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.634868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kserve-provision-location\") pod \"isvc-sklearn-predictor-5bd8597c94-n9d8f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.736026 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.735965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5bd8597c94-n9d8f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.736026 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.736015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41329f3e-36dd-45f3-8595-85cfb0db2a3f-proxy-tls\") pod \"isvc-sklearn-predictor-5bd8597c94-n9d8f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.736154 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.736047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28bqz\" (UniqueName: \"kubernetes.io/projected/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kube-api-access-28bqz\") pod \"isvc-sklearn-predictor-5bd8597c94-n9d8f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.736154 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.736086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kserve-provision-location\") pod \"isvc-sklearn-predictor-5bd8597c94-n9d8f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.736154 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.736111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41329f3e-36dd-45f3-8595-85cfb0db2a3f-cabundle-cert\") pod \"isvc-sklearn-predictor-5bd8597c94-n9d8f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.736529 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.736509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kserve-provision-location\") pod \"isvc-sklearn-predictor-5bd8597c94-n9d8f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.736762 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.736716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5bd8597c94-n9d8f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.736762 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.736716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41329f3e-36dd-45f3-8595-85cfb0db2a3f-cabundle-cert\") pod \"isvc-sklearn-predictor-5bd8597c94-n9d8f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.738451 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.738432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41329f3e-36dd-45f3-8595-85cfb0db2a3f-proxy-tls\") pod \"isvc-sklearn-predictor-5bd8597c94-n9d8f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.747776 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.747756    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28bqz\" (UniqueName: \"kubernetes.io/projected/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kube-api-access-28bqz\") pod \"isvc-sklearn-predictor-5bd8597c94-n9d8f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:48.856249 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:48.856229    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:49.134984 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:49.134960    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"]
Sep 16 17:48:49.136911 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:48:49.136862    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41329f3e_36dd_45f3_8595_85cfb0db2a3f.slice/crio-724da691460ada5a55a19d365767b3c461ce40eda29641cbca7ec3749f0cc429 WatchSource:0}: Error finding container 724da691460ada5a55a19d365767b3c461ce40eda29641cbca7ec3749f0cc429: Status 404 returned error can't find the container with id 724da691460ada5a55a19d365767b3c461ce40eda29641cbca7ec3749f0cc429
Sep 16 17:48:49.879518 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:49.879483    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" event={"ID":"41329f3e-36dd-45f3-8595-85cfb0db2a3f","Type":"ContainerStarted","Data":"31c247ef8c7e7cd735360640fb64eb58e831b9c01c9094fa865fba7015956f11"}
Sep 16 17:48:49.879864 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:49.879543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" event={"ID":"41329f3e-36dd-45f3-8595-85cfb0db2a3f","Type":"ContainerStarted","Data":"724da691460ada5a55a19d365767b3c461ce40eda29641cbca7ec3749f0cc429"}
Sep 16 17:48:53.896923 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:53.896871    2581 generic.go:356] "Generic (PLEG): container finished" podID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerID="31c247ef8c7e7cd735360640fb64eb58e831b9c01c9094fa865fba7015956f11" exitCode=0
Sep 16 17:48:53.897272 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:53.896941    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" event={"ID":"41329f3e-36dd-45f3-8595-85cfb0db2a3f","Type":"ContainerDied","Data":"31c247ef8c7e7cd735360640fb64eb58e831b9c01c9094fa865fba7015956f11"}
Sep 16 17:48:54.903623 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:54.903587    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" event={"ID":"41329f3e-36dd-45f3-8595-85cfb0db2a3f","Type":"ContainerStarted","Data":"09aaed193a479bda8219621b358cda93039dc5687383a592b468f329b3446ebb"}
Sep 16 17:48:54.903988 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:54.903654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" event={"ID":"41329f3e-36dd-45f3-8595-85cfb0db2a3f","Type":"ContainerStarted","Data":"b569c41f44638c8cc51d99891cf0e09559e19a307822054d88d1f90633150bfe"}
Sep 16 17:48:54.903988 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:54.903839    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:54.929857 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:54.929813    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" podStartSLOduration=2.959230236 podStartE2EDuration="6.929800509s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.970570273s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:48:54.928756735 +0000 UTC m=+1463.717379268" watchObservedRunningTime="2026-09-16 17:48:54.929800509 +0000 UTC m=+1463.718423041"
Sep 16 17:48:55.907752 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:55.907716    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:48:55.908943 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:55.908920    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 17:48:56.911837 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:48:56.911792    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 17:49:01.916146 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:49:01.916119    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:49:01.916683 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:49:01.916655    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 17:49:11.916761 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:49:11.916723    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 17:49:21.916951 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:49:21.916915    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 17:49:31.776271 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:49:31.776245    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:49:31.777612 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:49:31.777587    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:49:31.917220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:49:31.917193    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 17:49:41.917919 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:49:41.917082    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 17:49:51.916590 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:49:51.916554    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 17:50:01.917576 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:01.917502    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:50:05.080941 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:05.080880    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:08.545016 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:08.544984    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"]
Sep 16 17:50:08.545380 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:08.545272    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kserve-container" containerID="cri-o://b569c41f44638c8cc51d99891cf0e09559e19a307822054d88d1f90633150bfe" gracePeriod=30
Sep 16 17:50:08.545380 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:08.545361    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kube-rbac-proxy" containerID="cri-o://09aaed193a479bda8219621b358cda93039dc5687383a592b468f329b3446ebb" gracePeriod=30
Sep 16 17:50:08.558779 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:08.558761    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:09.194017 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:09.193985    2581 generic.go:356] "Generic (PLEG): container finished" podID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerID="09aaed193a479bda8219621b358cda93039dc5687383a592b468f329b3446ebb" exitCode=2
Sep 16 17:50:09.194154 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:09.194057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" event={"ID":"41329f3e-36dd-45f3-8595-85cfb0db2a3f","Type":"ContainerDied","Data":"09aaed193a479bda8219621b358cda93039dc5687383a592b468f329b3446ebb"}
Sep 16 17:50:10.176047 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:10.175967    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:10.348032 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:10.347998    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:11.912236 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:11.912203    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 16 17:50:11.917042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:11.917018    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 17:50:12.214544 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.214402    2581 generic.go:356] "Generic (PLEG): container finished" podID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerID="b569c41f44638c8cc51d99891cf0e09559e19a307822054d88d1f90633150bfe" exitCode=0
Sep 16 17:50:12.214544 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.214444    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" event={"ID":"41329f3e-36dd-45f3-8595-85cfb0db2a3f","Type":"ContainerDied","Data":"b569c41f44638c8cc51d99891cf0e09559e19a307822054d88d1f90633150bfe"}
Sep 16 17:50:12.287701 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.287676    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:50:12.314314 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.314291    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41329f3e-36dd-45f3-8595-85cfb0db2a3f-cabundle-cert\") pod \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") "
Sep 16 17:50:12.314419 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.314322    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kube-rbac-proxy-sar-config\") pod \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") "
Sep 16 17:50:12.314419 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.314344    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28bqz\" (UniqueName: \"kubernetes.io/projected/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kube-api-access-28bqz\") pod \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") "
Sep 16 17:50:12.314419 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.314391    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kserve-provision-location\") pod \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") "
Sep 16 17:50:12.314419 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.314415    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41329f3e-36dd-45f3-8595-85cfb0db2a3f-proxy-tls\") pod \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\" (UID: \"41329f3e-36dd-45f3-8595-85cfb0db2a3f\") "
Sep 16 17:50:12.314741 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.314715    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41329f3e-36dd-45f3-8595-85cfb0db2a3f" (UID: "41329f3e-36dd-45f3-8595-85cfb0db2a3f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:50:12.314874 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.314774    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41329f3e-36dd-45f3-8595-85cfb0db2a3f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41329f3e-36dd-45f3-8595-85cfb0db2a3f" (UID: "41329f3e-36dd-45f3-8595-85cfb0db2a3f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:50:12.314874 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.314802    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41329f3e-36dd-45f3-8595-85cfb0db2a3f" (UID: "41329f3e-36dd-45f3-8595-85cfb0db2a3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:50:12.316606 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.316586    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kube-api-access-28bqz" (OuterVolumeSpecName: "kube-api-access-28bqz") pod "41329f3e-36dd-45f3-8595-85cfb0db2a3f" (UID: "41329f3e-36dd-45f3-8595-85cfb0db2a3f"). InnerVolumeSpecName "kube-api-access-28bqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:50:12.316701 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.316665    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41329f3e-36dd-45f3-8595-85cfb0db2a3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41329f3e-36dd-45f3-8595-85cfb0db2a3f" (UID: "41329f3e-36dd-45f3-8595-85cfb0db2a3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:50:12.415620 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.415564    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41329f3e-36dd-45f3-8595-85cfb0db2a3f-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:50:12.415620 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.415586    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:50:12.415620 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.415595    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28bqz\" (UniqueName: \"kubernetes.io/projected/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kube-api-access-28bqz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:50:12.415620 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.415605    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41329f3e-36dd-45f3-8595-85cfb0db2a3f-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:50:12.415620 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:12.415614    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41329f3e-36dd-45f3-8595-85cfb0db2a3f-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:50:13.219973 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.219947    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f" event={"ID":"41329f3e-36dd-45f3-8595-85cfb0db2a3f","Type":"ContainerDied","Data":"724da691460ada5a55a19d365767b3c461ce40eda29641cbca7ec3749f0cc429"}
Sep 16 17:50:13.220293 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.219989    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"
Sep 16 17:50:13.220293 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.220004    2581 scope.go:117] "RemoveContainer" containerID="09aaed193a479bda8219621b358cda93039dc5687383a592b468f329b3446ebb"
Sep 16 17:50:13.228616 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.228601    2581 scope.go:117] "RemoveContainer" containerID="b569c41f44638c8cc51d99891cf0e09559e19a307822054d88d1f90633150bfe"
Sep 16 17:50:13.235902 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.235873    2581 scope.go:117] "RemoveContainer" containerID="31c247ef8c7e7cd735360640fb64eb58e831b9c01c9094fa865fba7015956f11"
Sep 16 17:50:13.243180 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.243159    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"]
Sep 16 17:50:13.249779 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.249763    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-n9d8f"]
Sep 16 17:50:13.649267 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.649239    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:13.676181 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.676155    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"]
Sep 16 17:50:13.676490 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.676477    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kserve-container"
Sep 16 17:50:13.676533 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.676492    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kserve-container"
Sep 16 17:50:13.676533 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.676508    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="storage-initializer"
Sep 16 17:50:13.676533 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.676514    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="storage-initializer"
Sep 16 17:50:13.676533 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.676520    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kube-rbac-proxy"
Sep 16 17:50:13.676533 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.676526    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kube-rbac-proxy"
Sep 16 17:50:13.676707 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.676584    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kserve-container"
Sep 16 17:50:13.676707 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.676593    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" containerName="kube-rbac-proxy"
Sep 16 17:50:13.681069 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.681046    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.684009 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.683982    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 17:50:13.684195 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.684071    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\""
Sep 16 17:50:13.684195 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.684071    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 17:50:13.684300 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.684203    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 17:50:13.685265 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.685247    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:50:13.685377 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.685338    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 16 17:50:13.685377 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.685364    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 17:50:13.691449 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.691429    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"]
Sep 16 17:50:13.724427 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.724403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b915d60e-a72c-4f70-a984-beaa00b9958b-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.724502 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.724441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b915d60e-a72c-4f70-a984-beaa00b9958b-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.724502 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.724464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ztk\" (UniqueName: \"kubernetes.io/projected/b915d60e-a72c-4f70-a984-beaa00b9958b-kube-api-access-x6ztk\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.724502 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.724479    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b915d60e-a72c-4f70-a984-beaa00b9958b-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.724608 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.724502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b915d60e-a72c-4f70-a984-beaa00b9958b-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.763400 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.763373    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:13.765765 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.765744    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41329f3e-36dd-45f3-8595-85cfb0db2a3f" path="/var/lib/kubelet/pods/41329f3e-36dd-45f3-8595-85cfb0db2a3f/volumes"
Sep 16 17:50:13.825649 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.825627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b915d60e-a72c-4f70-a984-beaa00b9958b-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.825742 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.825663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b915d60e-a72c-4f70-a984-beaa00b9958b-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.825742 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.825688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ztk\" (UniqueName: \"kubernetes.io/projected/b915d60e-a72c-4f70-a984-beaa00b9958b-kube-api-access-x6ztk\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.825742 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.825706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b915d60e-a72c-4f70-a984-beaa00b9958b-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.825742 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.825733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b915d60e-a72c-4f70-a984-beaa00b9958b-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.826171 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.826151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b915d60e-a72c-4f70-a984-beaa00b9958b-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.826292 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.826273    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b915d60e-a72c-4f70-a984-beaa00b9958b-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.826363 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.826349    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b915d60e-a72c-4f70-a984-beaa00b9958b-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.828182 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.828147    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b915d60e-a72c-4f70-a984-beaa00b9958b-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.839153 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.839132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ztk\" (UniqueName: \"kubernetes.io/projected/b915d60e-a72c-4f70-a984-beaa00b9958b-kube-api-access-x6ztk\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:13.991428 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:13.991369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:50:14.263361 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:14.263340    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"]
Sep 16 17:50:14.265150 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:50:14.265119    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb915d60e_a72c_4f70_a984_beaa00b9958b.slice/crio-bf18c1fd3d2e2459484b2f9b410817b3439e2185ad7b9dce3573d7d634b21441 WatchSource:0}: Error finding container bf18c1fd3d2e2459484b2f9b410817b3439e2185ad7b9dce3573d7d634b21441: Status 404 returned error can't find the container with id bf18c1fd3d2e2459484b2f9b410817b3439e2185ad7b9dce3573d7d634b21441
Sep 16 17:50:15.231115 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:15.231084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d" event={"ID":"b915d60e-a72c-4f70-a984-beaa00b9958b","Type":"ContainerStarted","Data":"3f81eb6cc0bc060fa70366c64c41113fb245bd1ba042157c06fdef5aa57cbcf9"}
Sep 16 17:50:15.231267 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:15.231135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d" event={"ID":"b915d60e-a72c-4f70-a984-beaa00b9958b","Type":"ContainerStarted","Data":"bf18c1fd3d2e2459484b2f9b410817b3439e2185ad7b9dce3573d7d634b21441"}
Sep 16 17:50:18.244636 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:18.244610    2581 generic.go:356] "Generic (PLEG): container finished" podID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerID="3f81eb6cc0bc060fa70366c64c41113fb245bd1ba042157c06fdef5aa57cbcf9" exitCode=0
Sep 16 17:50:18.245049 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:50:18.244683    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d" event={"ID":"b915d60e-a72c-4f70-a984-beaa00b9958b","Type":"ContainerDied","Data":"3f81eb6cc0bc060fa70366c64c41113fb245bd1ba042157c06fdef5aa57cbcf9"}
Sep 16 17:51:12.063368 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:51:12.063334    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:52:17.822448 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:17.822411    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d" event={"ID":"b915d60e-a72c-4f70-a984-beaa00b9958b","Type":"ContainerStarted","Data":"695a33f1aa9518d8893b06a6c693243a410181cca1a7774ad85faeb387852219"}
Sep 16 17:52:17.822879 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:17.822478    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d" event={"ID":"b915d60e-a72c-4f70-a984-beaa00b9958b","Type":"ContainerStarted","Data":"6a582528f893318721dce03d3a60b40a45a4eb0e979cc60ed55be97fcba5f101"}
Sep 16 17:52:17.822879 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:17.822511    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:52:17.822879 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:17.822541    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:52:17.847010 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:17.846965    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d" podStartSLOduration=2.155239414 podStartE2EDuration="2m4.84695013s" totalImagesPullingTime="1m58.923189676s" totalInitContainerRuntime="3.76852104s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:52:17.845094675 +0000 UTC m=+1666.633717208" watchObservedRunningTime="2026-09-16 17:52:17.84695013 +0000 UTC m=+1666.635572659"
Sep 16 17:52:23.831664 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:23.831635    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:52:28.097539 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:28.097509    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:52:31.835792 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:31.835762    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:52:31.905814 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:31.905782    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"]
Sep 16 17:52:31.912903 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:31.912878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:31.916156 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:31.916136    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 16 17:52:31.916272 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:31.916171    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 16 17:52:31.922778 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:31.922757    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"]
Sep 16 17:52:32.015781 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.015755    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:52:32.039412 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.039392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmw6k\" (UniqueName: \"kubernetes.io/projected/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kube-api-access-pmw6k\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.039505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.039434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.039505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.039455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.039581 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.039504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.039581 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.039530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.139871 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.139805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmw6k\" (UniqueName: \"kubernetes.io/projected/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kube-api-access-pmw6k\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.139871 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.139856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.140040 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.139884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.140040 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.139919    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.140040 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.139943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.140457 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.140436    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.140724 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.140637    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.140724 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.140664    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.142514 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.142492    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.149270 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.149247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmw6k\" (UniqueName: \"kubernetes.io/projected/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kube-api-access-pmw6k\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.224174 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.224152    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:32.499677 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.499653    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"]
Sep 16 17:52:32.501365 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:52:32.501335    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ba77ba_634e_4f79_8ab3_4d3a2d3a0d09.slice/crio-78ee0f4c6ec00c490acb59fa5c2deec4df3671dee392241033ada764e32e7754 WatchSource:0}: Error finding container 78ee0f4c6ec00c490acb59fa5c2deec4df3671dee392241033ada764e32e7754: Status 404 returned error can't find the container with id 78ee0f4c6ec00c490acb59fa5c2deec4df3671dee392241033ada764e32e7754
Sep 16 17:52:32.882645 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.882609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd" event={"ID":"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09","Type":"ContainerStarted","Data":"eade65062087b035f8b2b8dcacee9acf1f32206126021723bdd57057c2d08222"}
Sep 16 17:52:32.883039 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:32.882668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd" event={"ID":"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09","Type":"ContainerStarted","Data":"78ee0f4c6ec00c490acb59fa5c2deec4df3671dee392241033ada764e32e7754"}
Sep 16 17:52:37.907198 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:37.907122    2581 generic.go:356] "Generic (PLEG): container finished" podID="f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" containerID="eade65062087b035f8b2b8dcacee9acf1f32206126021723bdd57057c2d08222" exitCode=0
Sep 16 17:52:37.907588 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:37.907194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd" event={"ID":"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09","Type":"ContainerDied","Data":"eade65062087b035f8b2b8dcacee9acf1f32206126021723bdd57057c2d08222"}
Sep 16 17:52:38.914805 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:38.914771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd" event={"ID":"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09","Type":"ContainerStarted","Data":"208ca6fb27d22c138f7ee1c1640cde613a4c9eba90f3f4c5211b028b7e471c70"}
Sep 16 17:52:38.915191 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:38.914833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd" event={"ID":"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09","Type":"ContainerStarted","Data":"96ea29857251e553a92ca3479d30b82000a7a8a6c8ee313825427ddc0fe4a135"}
Sep 16 17:52:38.915243 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:38.915187    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:38.915243 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:38.915236    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:38.938298 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:38.938257    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd" podStartSLOduration=2.8699586889999997 podStartE2EDuration="7.938245093s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.068286404s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:52:38.936940048 +0000 UTC m=+1687.725562584" watchObservedRunningTime="2026-09-16 17:52:38.938245093 +0000 UTC m=+1687.726867625"
Sep 16 17:52:44.924823 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:44.924793    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:52:53.859878 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:52:53.859792    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d" podUID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 17:53:03.834262 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:03.834236    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:53:13.801623 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:13.801590    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"]
Sep 16 17:53:13.802075 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:13.801868    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d" podUID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerName="kserve-container" containerID="cri-o://6a582528f893318721dce03d3a60b40a45a4eb0e979cc60ed55be97fcba5f101" gracePeriod=30
Sep 16 17:53:13.802075 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:13.801960    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d" podUID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerName="kube-rbac-proxy" containerID="cri-o://695a33f1aa9518d8893b06a6c693243a410181cca1a7774ad85faeb387852219" gracePeriod=30
Sep 16 17:53:13.817443 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:13.817423    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:13.826787 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:13.826746    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d" podUID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 16 17:53:14.061604 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:14.061547    2581 generic.go:356] "Generic (PLEG): container finished" podID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerID="695a33f1aa9518d8893b06a6c693243a410181cca1a7774ad85faeb387852219" exitCode=2
Sep 16 17:53:14.061715 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:14.061626    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d" event={"ID":"b915d60e-a72c-4f70-a984-beaa00b9958b","Type":"ContainerDied","Data":"695a33f1aa9518d8893b06a6c693243a410181cca1a7774ad85faeb387852219"}
Sep 16 17:53:14.873045 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:14.873001    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d" podUID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 17:53:18.827076 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:18.827043    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d" podUID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 16 17:53:19.023018 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.022992    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:19.073269 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.073239    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"]
Sep 16 17:53:19.076748 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.076727    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.079790 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.079723    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 16 17:53:19.080223 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.080187    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 16 17:53:19.090457 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.090429    2581 generic.go:356] "Generic (PLEG): container finished" podID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerID="6a582528f893318721dce03d3a60b40a45a4eb0e979cc60ed55be97fcba5f101" exitCode=0
Sep 16 17:53:19.090574 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.090507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d" event={"ID":"b915d60e-a72c-4f70-a984-beaa00b9958b","Type":"ContainerDied","Data":"6a582528f893318721dce03d3a60b40a45a4eb0e979cc60ed55be97fcba5f101"}
Sep 16 17:53:19.090882 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.090863    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"]
Sep 16 17:53:19.113181 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.113161    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:19.138488 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.138472    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:53:19.187376 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.187355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.187490 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.187413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f470e5a2-223d-45bb-adcd-9ae82852a6a6-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.187557 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.187499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f470e5a2-223d-45bb-adcd-9ae82852a6a6-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.187557 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.187535    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.187670 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.187609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcfhk\" (UniqueName: \"kubernetes.io/projected/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kube-api-access-pcfhk\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.288752 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.288729    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6ztk\" (UniqueName: \"kubernetes.io/projected/b915d60e-a72c-4f70-a984-beaa00b9958b-kube-api-access-x6ztk\") pod \"b915d60e-a72c-4f70-a984-beaa00b9958b\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") "
Sep 16 17:53:19.288854 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.288765    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b915d60e-a72c-4f70-a984-beaa00b9958b-kserve-provision-location\") pod \"b915d60e-a72c-4f70-a984-beaa00b9958b\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") "
Sep 16 17:53:19.288854 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.288796    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b915d60e-a72c-4f70-a984-beaa00b9958b-kube-rbac-proxy-sar-config\") pod \"b915d60e-a72c-4f70-a984-beaa00b9958b\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") "
Sep 16 17:53:19.288854 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.288825    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b915d60e-a72c-4f70-a984-beaa00b9958b-proxy-tls\") pod \"b915d60e-a72c-4f70-a984-beaa00b9958b\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") "
Sep 16 17:53:19.289054 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.288882    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b915d60e-a72c-4f70-a984-beaa00b9958b-cabundle-cert\") pod \"b915d60e-a72c-4f70-a984-beaa00b9958b\" (UID: \"b915d60e-a72c-4f70-a984-beaa00b9958b\") "
Sep 16 17:53:19.289111 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.289058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f470e5a2-223d-45bb-adcd-9ae82852a6a6-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.289111 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.289101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f470e5a2-223d-45bb-adcd-9ae82852a6a6-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.289219 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.289127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.289219 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.289130    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b915d60e-a72c-4f70-a984-beaa00b9958b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b915d60e-a72c-4f70-a984-beaa00b9958b" (UID: "b915d60e-a72c-4f70-a984-beaa00b9958b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:53:19.289219 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.289173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcfhk\" (UniqueName: \"kubernetes.io/projected/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kube-api-access-pcfhk\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.289368 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.289221    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b915d60e-a72c-4f70-a984-beaa00b9958b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b915d60e-a72c-4f70-a984-beaa00b9958b" (UID: "b915d60e-a72c-4f70-a984-beaa00b9958b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:53:19.289368 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.289233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.289368 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.289292    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b915d60e-a72c-4f70-a984-beaa00b9958b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b915d60e-a72c-4f70-a984-beaa00b9958b" (UID: "b915d60e-a72c-4f70-a984-beaa00b9958b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:53:19.289368 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.289305    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b915d60e-a72c-4f70-a984-beaa00b9958b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:53:19.289368 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.289320    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b915d60e-a72c-4f70-a984-beaa00b9958b-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:53:19.289830 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.289806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.289926 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.289826    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f470e5a2-223d-45bb-adcd-9ae82852a6a6-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.289992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.289939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.291101 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.291081    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b915d60e-a72c-4f70-a984-beaa00b9958b-kube-api-access-x6ztk" (OuterVolumeSpecName: "kube-api-access-x6ztk") pod "b915d60e-a72c-4f70-a984-beaa00b9958b" (UID: "b915d60e-a72c-4f70-a984-beaa00b9958b"). InnerVolumeSpecName "kube-api-access-x6ztk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:53:19.291176 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.291169    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b915d60e-a72c-4f70-a984-beaa00b9958b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b915d60e-a72c-4f70-a984-beaa00b9958b" (UID: "b915d60e-a72c-4f70-a984-beaa00b9958b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:53:19.291903 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.291873    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f470e5a2-223d-45bb-adcd-9ae82852a6a6-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.300712 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.300690    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcfhk\" (UniqueName: \"kubernetes.io/projected/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kube-api-access-pcfhk\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.389399 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.389354    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:19.389630 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.389613    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b915d60e-a72c-4f70-a984-beaa00b9958b-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:53:19.389674 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.389634    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b915d60e-a72c-4f70-a984-beaa00b9958b-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:53:19.389674 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.389643    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6ztk\" (UniqueName: \"kubernetes.io/projected/b915d60e-a72c-4f70-a984-beaa00b9958b-kube-api-access-x6ztk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:53:19.662406 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:19.662380    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"]
Sep 16 17:53:19.663874 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:53:19.663846    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf470e5a2_223d_45bb_adcd_9ae82852a6a6.slice/crio-d5d480a55e8e217c0e841b86343bd56528e100483f1e0e92266ab9f474b819a6 WatchSource:0}: Error finding container d5d480a55e8e217c0e841b86343bd56528e100483f1e0e92266ab9f474b819a6: Status 404 returned error can't find the container with id d5d480a55e8e217c0e841b86343bd56528e100483f1e0e92266ab9f474b819a6
Sep 16 17:53:20.097368 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:20.097333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs" event={"ID":"f470e5a2-223d-45bb-adcd-9ae82852a6a6","Type":"ContainerStarted","Data":"e785da4bb14b5a01407589fcafb576ffc733c016732923a75cbffaaec7c02bbb"}
Sep 16 17:53:20.097777 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:20.097393    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs" event={"ID":"f470e5a2-223d-45bb-adcd-9ae82852a6a6","Type":"ContainerStarted","Data":"d5d480a55e8e217c0e841b86343bd56528e100483f1e0e92266ab9f474b819a6"}
Sep 16 17:53:20.099392 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:20.099365    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d" event={"ID":"b915d60e-a72c-4f70-a984-beaa00b9958b","Type":"ContainerDied","Data":"bf18c1fd3d2e2459484b2f9b410817b3439e2185ad7b9dce3573d7d634b21441"}
Sep 16 17:53:20.099392 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:20.099388    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"
Sep 16 17:53:20.099559 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:20.099421    2581 scope.go:117] "RemoveContainer" containerID="695a33f1aa9518d8893b06a6c693243a410181cca1a7774ad85faeb387852219"
Sep 16 17:53:20.107369 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:20.107349    2581 scope.go:117] "RemoveContainer" containerID="6a582528f893318721dce03d3a60b40a45a4eb0e979cc60ed55be97fcba5f101"
Sep 16 17:53:20.114539 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:20.114517    2581 scope.go:117] "RemoveContainer" containerID="3f81eb6cc0bc060fa70366c64c41113fb245bd1ba042157c06fdef5aa57cbcf9"
Sep 16 17:53:20.133935 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:20.133911    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"]
Sep 16 17:53:20.137825 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:20.137802    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-5rv6d"]
Sep 16 17:53:21.766461 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:21.766430    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b915d60e-a72c-4f70-a984-beaa00b9958b" path="/var/lib/kubelet/pods/b915d60e-a72c-4f70-a984-beaa00b9958b/volumes"
Sep 16 17:53:25.124849 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:25.124817    2581 generic.go:356] "Generic (PLEG): container finished" podID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerID="e785da4bb14b5a01407589fcafb576ffc733c016732923a75cbffaaec7c02bbb" exitCode=0
Sep 16 17:53:25.125220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:25.124855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs" event={"ID":"f470e5a2-223d-45bb-adcd-9ae82852a6a6","Type":"ContainerDied","Data":"e785da4bb14b5a01407589fcafb576ffc733c016732923a75cbffaaec7c02bbb"}
Sep 16 17:53:26.133628 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:26.133592    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs" event={"ID":"f470e5a2-223d-45bb-adcd-9ae82852a6a6","Type":"ContainerStarted","Data":"379c5e0d1019eb348d10d38a02185b6e35d804b2ea6eb54a366e3fdaf1d49d62"}
Sep 16 17:53:26.133998 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:26.133674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs" event={"ID":"f470e5a2-223d-45bb-adcd-9ae82852a6a6","Type":"ContainerStarted","Data":"fcbc7b6029c13af77b6f0c314986f4ac4cd4d8dc31a544bfc64a1ca9f9258615"}
Sep 16 17:53:26.134067 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:26.134005    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:26.134067 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:26.134059    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:26.135389 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:26.135362    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs" podUID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 17:53:26.158372 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:26.158320    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs" podStartSLOduration=1.808413619 podStartE2EDuration="7.158301253s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.349887634s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:53:26.155727915 +0000 UTC m=+1734.944350451" watchObservedRunningTime="2026-09-16 17:53:26.158301253 +0000 UTC m=+1734.946923789"
Sep 16 17:53:27.139092 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:27.139049    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs" podUID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 17:53:30.177616 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:30.177586    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:32.143953 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:32.143925    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:32.144629 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:32.144592    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs" podUID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 17:53:42.145287 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:42.145259    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:51.020526 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:51.020494    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"]
Sep 16 17:53:51.020990 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:51.020704    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs" podUID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerName="kserve-container" containerID="cri-o://fcbc7b6029c13af77b6f0c314986f4ac4cd4d8dc31a544bfc64a1ca9f9258615" gracePeriod=30
Sep 16 17:53:51.020990 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:51.020788    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs" podUID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerName="kube-rbac-proxy" containerID="cri-o://379c5e0d1019eb348d10d38a02185b6e35d804b2ea6eb54a366e3fdaf1d49d62" gracePeriod=30
Sep 16 17:53:51.040512 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:51.040491    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:51.258254 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:51.258223    2581 generic.go:356] "Generic (PLEG): container finished" podID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerID="379c5e0d1019eb348d10d38a02185b6e35d804b2ea6eb54a366e3fdaf1d49d62" exitCode=2
Sep 16 17:53:51.258395 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:51.258299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs" event={"ID":"f470e5a2-223d-45bb-adcd-9ae82852a6a6","Type":"ContainerDied","Data":"379c5e0d1019eb348d10d38a02185b6e35d804b2ea6eb54a366e3fdaf1d49d62"}
Sep 16 17:53:51.967130 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:51.967105    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:52.028027 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.028003    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f470e5a2-223d-45bb-adcd-9ae82852a6a6-proxy-tls\") pod \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") "
Sep 16 17:53:52.028369 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.028032    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kserve-provision-location\") pod \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") "
Sep 16 17:53:52.028369 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.028067    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f470e5a2-223d-45bb-adcd-9ae82852a6a6-cabundle-cert\") pod \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") "
Sep 16 17:53:52.028369 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.028112    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kube-rbac-proxy-sar-config\") pod \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") "
Sep 16 17:53:52.028369 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.028159    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcfhk\" (UniqueName: \"kubernetes.io/projected/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kube-api-access-pcfhk\") pod \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\" (UID: \"f470e5a2-223d-45bb-adcd-9ae82852a6a6\") "
Sep 16 17:53:52.028572 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.028501    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f470e5a2-223d-45bb-adcd-9ae82852a6a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f470e5a2-223d-45bb-adcd-9ae82852a6a6" (UID: "f470e5a2-223d-45bb-adcd-9ae82852a6a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:53:52.028572 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.028507    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f470e5a2-223d-45bb-adcd-9ae82852a6a6" (UID: "f470e5a2-223d-45bb-adcd-9ae82852a6a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:53:52.030140 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.030120    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f470e5a2-223d-45bb-adcd-9ae82852a6a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f470e5a2-223d-45bb-adcd-9ae82852a6a6" (UID: "f470e5a2-223d-45bb-adcd-9ae82852a6a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:53:52.030324 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.030303    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kube-api-access-pcfhk" (OuterVolumeSpecName: "kube-api-access-pcfhk") pod "f470e5a2-223d-45bb-adcd-9ae82852a6a6" (UID: "f470e5a2-223d-45bb-adcd-9ae82852a6a6"). InnerVolumeSpecName "kube-api-access-pcfhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:53:52.058492 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.058466    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f470e5a2-223d-45bb-adcd-9ae82852a6a6" (UID: "f470e5a2-223d-45bb-adcd-9ae82852a6a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:53:52.129535 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.129485    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f470e5a2-223d-45bb-adcd-9ae82852a6a6-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:53:52.129535 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.129509    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:53:52.129535 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.129518    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f470e5a2-223d-45bb-adcd-9ae82852a6a6-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:53:52.129535 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.129527    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:53:52.129535 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.129535    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcfhk\" (UniqueName: \"kubernetes.io/projected/f470e5a2-223d-45bb-adcd-9ae82852a6a6-kube-api-access-pcfhk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:53:52.263394 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.263369    2581 generic.go:356] "Generic (PLEG): container finished" podID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerID="fcbc7b6029c13af77b6f0c314986f4ac4cd4d8dc31a544bfc64a1ca9f9258615" exitCode=0
Sep 16 17:53:52.263482 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.263429    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs" event={"ID":"f470e5a2-223d-45bb-adcd-9ae82852a6a6","Type":"ContainerDied","Data":"fcbc7b6029c13af77b6f0c314986f4ac4cd4d8dc31a544bfc64a1ca9f9258615"}
Sep 16 17:53:52.263482 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.263438    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"
Sep 16 17:53:52.263560 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.263514    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs" event={"ID":"f470e5a2-223d-45bb-adcd-9ae82852a6a6","Type":"ContainerDied","Data":"d5d480a55e8e217c0e841b86343bd56528e100483f1e0e92266ab9f474b819a6"}
Sep 16 17:53:52.263560 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.263552    2581 scope.go:117] "RemoveContainer" containerID="379c5e0d1019eb348d10d38a02185b6e35d804b2ea6eb54a366e3fdaf1d49d62"
Sep 16 17:53:52.271745 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.271730    2581 scope.go:117] "RemoveContainer" containerID="fcbc7b6029c13af77b6f0c314986f4ac4cd4d8dc31a544bfc64a1ca9f9258615"
Sep 16 17:53:52.278671 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.278653    2581 scope.go:117] "RemoveContainer" containerID="e785da4bb14b5a01407589fcafb576ffc733c016732923a75cbffaaec7c02bbb"
Sep 16 17:53:52.285800 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.285783    2581 scope.go:117] "RemoveContainer" containerID="379c5e0d1019eb348d10d38a02185b6e35d804b2ea6eb54a366e3fdaf1d49d62"
Sep 16 17:53:52.286086 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:53:52.286066    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379c5e0d1019eb348d10d38a02185b6e35d804b2ea6eb54a366e3fdaf1d49d62\": container with ID starting with 379c5e0d1019eb348d10d38a02185b6e35d804b2ea6eb54a366e3fdaf1d49d62 not found: ID does not exist" containerID="379c5e0d1019eb348d10d38a02185b6e35d804b2ea6eb54a366e3fdaf1d49d62"
Sep 16 17:53:52.286173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.286091    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379c5e0d1019eb348d10d38a02185b6e35d804b2ea6eb54a366e3fdaf1d49d62"} err="failed to get container status \"379c5e0d1019eb348d10d38a02185b6e35d804b2ea6eb54a366e3fdaf1d49d62\": rpc error: code = NotFound desc = could not find container \"379c5e0d1019eb348d10d38a02185b6e35d804b2ea6eb54a366e3fdaf1d49d62\": container with ID starting with 379c5e0d1019eb348d10d38a02185b6e35d804b2ea6eb54a366e3fdaf1d49d62 not found: ID does not exist"
Sep 16 17:53:52.286173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.286107    2581 scope.go:117] "RemoveContainer" containerID="fcbc7b6029c13af77b6f0c314986f4ac4cd4d8dc31a544bfc64a1ca9f9258615"
Sep 16 17:53:52.286331 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:53:52.286317    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcbc7b6029c13af77b6f0c314986f4ac4cd4d8dc31a544bfc64a1ca9f9258615\": container with ID starting with fcbc7b6029c13af77b6f0c314986f4ac4cd4d8dc31a544bfc64a1ca9f9258615 not found: ID does not exist" containerID="fcbc7b6029c13af77b6f0c314986f4ac4cd4d8dc31a544bfc64a1ca9f9258615"
Sep 16 17:53:52.286370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.286334    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcbc7b6029c13af77b6f0c314986f4ac4cd4d8dc31a544bfc64a1ca9f9258615"} err="failed to get container status \"fcbc7b6029c13af77b6f0c314986f4ac4cd4d8dc31a544bfc64a1ca9f9258615\": rpc error: code = NotFound desc = could not find container \"fcbc7b6029c13af77b6f0c314986f4ac4cd4d8dc31a544bfc64a1ca9f9258615\": container with ID starting with fcbc7b6029c13af77b6f0c314986f4ac4cd4d8dc31a544bfc64a1ca9f9258615 not found: ID does not exist"
Sep 16 17:53:52.286370 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.286346    2581 scope.go:117] "RemoveContainer" containerID="e785da4bb14b5a01407589fcafb576ffc733c016732923a75cbffaaec7c02bbb"
Sep 16 17:53:52.286567 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:53:52.286553    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e785da4bb14b5a01407589fcafb576ffc733c016732923a75cbffaaec7c02bbb\": container with ID starting with e785da4bb14b5a01407589fcafb576ffc733c016732923a75cbffaaec7c02bbb not found: ID does not exist" containerID="e785da4bb14b5a01407589fcafb576ffc733c016732923a75cbffaaec7c02bbb"
Sep 16 17:53:52.286610 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.286569    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e785da4bb14b5a01407589fcafb576ffc733c016732923a75cbffaaec7c02bbb"} err="failed to get container status \"e785da4bb14b5a01407589fcafb576ffc733c016732923a75cbffaaec7c02bbb\": rpc error: code = NotFound desc = could not find container \"e785da4bb14b5a01407589fcafb576ffc733c016732923a75cbffaaec7c02bbb\": container with ID starting with e785da4bb14b5a01407589fcafb576ffc733c016732923a75cbffaaec7c02bbb not found: ID does not exist"
Sep 16 17:53:52.287089 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.287072    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"]
Sep 16 17:53:52.294212 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:52.294186    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-c8jvs"]
Sep 16 17:53:53.766234 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:53.766205    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" path="/var/lib/kubelet/pods/f470e5a2-223d-45bb-adcd-9ae82852a6a6/volumes"
Sep 16 17:53:56.005756 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.005726    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:56.044358 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044329    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"]
Sep 16 17:53:56.044627 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044616    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerName="kserve-container"
Sep 16 17:53:56.044678 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044629    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerName="kserve-container"
Sep 16 17:53:56.044678 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044639    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerName="kube-rbac-proxy"
Sep 16 17:53:56.044678 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044644    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerName="kube-rbac-proxy"
Sep 16 17:53:56.044678 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044655    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerName="storage-initializer"
Sep 16 17:53:56.044678 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044661    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerName="storage-initializer"
Sep 16 17:53:56.044678 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044669    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerName="kserve-container"
Sep 16 17:53:56.044678 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044674    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerName="kserve-container"
Sep 16 17:53:56.044885 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044686    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerName="storage-initializer"
Sep 16 17:53:56.044885 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044691    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerName="storage-initializer"
Sep 16 17:53:56.044885 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044699    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerName="kube-rbac-proxy"
Sep 16 17:53:56.044885 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044703    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerName="kube-rbac-proxy"
Sep 16 17:53:56.044885 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044751    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerName="kube-rbac-proxy"
Sep 16 17:53:56.044885 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044759    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerName="kserve-container"
Sep 16 17:53:56.044885 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044767    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b915d60e-a72c-4f70-a984-beaa00b9958b" containerName="kserve-container"
Sep 16 17:53:56.044885 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.044773    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f470e5a2-223d-45bb-adcd-9ae82852a6a6" containerName="kube-rbac-proxy"
Sep 16 17:53:56.049246 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.049228    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.053394 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.053371    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 17:53:56.053589 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.053577    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 16 17:53:56.062035 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.062013    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"]
Sep 16 17:53:56.097802 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.097780    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:56.156153 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.156131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a6addb7-0378-4bf3-98b9-c98a665860c5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.156236 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.156179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a6addb7-0378-4bf3-98b9-c98a665860c5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.156236 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.156210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a6addb7-0378-4bf3-98b9-c98a665860c5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.156337 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.156240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a6addb7-0378-4bf3-98b9-c98a665860c5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.156337 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.156265    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-926xk\" (UniqueName: \"kubernetes.io/projected/8a6addb7-0378-4bf3-98b9-c98a665860c5-kube-api-access-926xk\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.256561 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.256512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-926xk\" (UniqueName: \"kubernetes.io/projected/8a6addb7-0378-4bf3-98b9-c98a665860c5-kube-api-access-926xk\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.256561 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.256547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a6addb7-0378-4bf3-98b9-c98a665860c5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.256722 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.256577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a6addb7-0378-4bf3-98b9-c98a665860c5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.256722 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.256597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a6addb7-0378-4bf3-98b9-c98a665860c5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.256722 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.256632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a6addb7-0378-4bf3-98b9-c98a665860c5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.257136 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.257112    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a6addb7-0378-4bf3-98b9-c98a665860c5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.257326 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.257306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a6addb7-0378-4bf3-98b9-c98a665860c5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.257326 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.257317    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a6addb7-0378-4bf3-98b9-c98a665860c5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.258923 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.258880    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a6addb7-0378-4bf3-98b9-c98a665860c5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.265972 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.265950    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-926xk\" (UniqueName: \"kubernetes.io/projected/8a6addb7-0378-4bf3-98b9-c98a665860c5-kube-api-access-926xk\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.359871 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.359837    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:53:56.631230 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:56.631206    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"]
Sep 16 17:53:56.633079 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:53:56.633050    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6addb7_0378_4bf3_98b9_c98a665860c5.slice/crio-15ace0c3083e0e857b7d6a1fc21a974c5272b7efc332f9659d7b604424412447 WatchSource:0}: Error finding container 15ace0c3083e0e857b7d6a1fc21a974c5272b7efc332f9659d7b604424412447: Status 404 returned error can't find the container with id 15ace0c3083e0e857b7d6a1fc21a974c5272b7efc332f9659d7b604424412447
Sep 16 17:53:57.286985 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:57.286938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw" event={"ID":"8a6addb7-0378-4bf3-98b9-c98a665860c5","Type":"ContainerStarted","Data":"caef26be73014f08f885a03f3dc3afaba97d55f87939181912197883c94f5a9e"}
Sep 16 17:53:57.287366 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:53:57.287020    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw" event={"ID":"8a6addb7-0378-4bf3-98b9-c98a665860c5","Type":"ContainerStarted","Data":"15ace0c3083e0e857b7d6a1fc21a974c5272b7efc332f9659d7b604424412447"}
Sep 16 17:54:01.305439 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:01.305405    2581 generic.go:356] "Generic (PLEG): container finished" podID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerID="caef26be73014f08f885a03f3dc3afaba97d55f87939181912197883c94f5a9e" exitCode=0
Sep 16 17:54:01.305798 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:01.305482    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw" event={"ID":"8a6addb7-0378-4bf3-98b9-c98a665860c5","Type":"ContainerDied","Data":"caef26be73014f08f885a03f3dc3afaba97d55f87939181912197883c94f5a9e"}
Sep 16 17:54:01.563704 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:01.563631    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:54:02.311867 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:02.311836    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw" event={"ID":"8a6addb7-0378-4bf3-98b9-c98a665860c5","Type":"ContainerStarted","Data":"2d0d38374e2eb6dd4adc231649962f567d56000cc47718125729e551de63f867"}
Sep 16 17:54:02.312253 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:02.311907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw" event={"ID":"8a6addb7-0378-4bf3-98b9-c98a665860c5","Type":"ContainerStarted","Data":"5d4538466ef4f12e9338e3a0b2b7defff950f5a273ec2f6a10b17122f740b70c"}
Sep 16 17:54:02.312253 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:02.312088    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:54:02.335943 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:02.335885    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw" podStartSLOduration=2.4048179259999998 podStartE2EDuration="6.335872009s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.931054083s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:54:02.334987506 +0000 UTC m=+1771.123610038" watchObservedRunningTime="2026-09-16 17:54:02.335872009 +0000 UTC m=+1771.124494541"
Sep 16 17:54:03.316551 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:03.316523    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:54:09.325562 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:09.325528    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:54:12.764862 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:12.764825    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:54:23.803118 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:23.803017    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd" podUID="f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 17:54:27.932277 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:27.932227    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"]
Sep 16 17:54:27.932621 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:27.932505    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd" podUID="f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" containerName="kserve-container" containerID="cri-o://96ea29857251e553a92ca3479d30b82000a7a8a6c8ee313825427ddc0fe4a135" gracePeriod=30
Sep 16 17:54:27.932621 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:27.932532    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd" podUID="f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" containerName="kube-rbac-proxy" containerID="cri-o://208ca6fb27d22c138f7ee1c1640cde613a4c9eba90f3f4c5211b028b7e471c70" gracePeriod=30
Sep 16 17:54:27.942878 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:27.942859    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:54:28.431115 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:28.431082    2581 generic.go:356] "Generic (PLEG): container finished" podID="f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" containerID="208ca6fb27d22c138f7ee1c1640cde613a4c9eba90f3f4c5211b028b7e471c70" exitCode=2
Sep 16 17:54:28.431310 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:28.431154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd" event={"ID":"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09","Type":"ContainerDied","Data":"208ca6fb27d22c138f7ee1c1640cde613a4c9eba90f3f4c5211b028b7e471c70"}
Sep 16 17:54:29.170207 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.170179    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:54:29.298932 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.298909    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kube-rbac-proxy-sar-config\") pod \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") "
Sep 16 17:54:29.299056 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.298945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmw6k\" (UniqueName: \"kubernetes.io/projected/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kube-api-access-pmw6k\") pod \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") "
Sep 16 17:54:29.299056 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.298979    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-cabundle-cert\") pod \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") "
Sep 16 17:54:29.299056 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.299033    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-proxy-tls\") pod \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") "
Sep 16 17:54:29.299056 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.299053    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kserve-provision-location\") pod \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\" (UID: \"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09\") "
Sep 16 17:54:29.299337 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.299315    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" (UID: "f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:54:29.299392 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.299346    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" (UID: "f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:54:29.301145 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.301121    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kube-api-access-pmw6k" (OuterVolumeSpecName: "kube-api-access-pmw6k") pod "f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" (UID: "f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09"). InnerVolumeSpecName "kube-api-access-pmw6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:54:29.301207 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.301190    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" (UID: "f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:54:29.303352 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.303329    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" (UID: "f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:54:29.399995 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.399973    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:54:29.399995 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.399992    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:54:29.400121 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.400001    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:54:29.400121 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.400011    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:54:29.400121 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.400020    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmw6k\" (UniqueName: \"kubernetes.io/projected/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09-kube-api-access-pmw6k\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:54:29.437647 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.437623    2581 generic.go:356] "Generic (PLEG): container finished" podID="f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" containerID="96ea29857251e553a92ca3479d30b82000a7a8a6c8ee313825427ddc0fe4a135" exitCode=0
Sep 16 17:54:29.437752 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.437689    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"
Sep 16 17:54:29.437752 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.437707    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd" event={"ID":"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09","Type":"ContainerDied","Data":"96ea29857251e553a92ca3479d30b82000a7a8a6c8ee313825427ddc0fe4a135"}
Sep 16 17:54:29.437869 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.437762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd" event={"ID":"f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09","Type":"ContainerDied","Data":"78ee0f4c6ec00c490acb59fa5c2deec4df3671dee392241033ada764e32e7754"}
Sep 16 17:54:29.437869 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.437790    2581 scope.go:117] "RemoveContainer" containerID="208ca6fb27d22c138f7ee1c1640cde613a4c9eba90f3f4c5211b028b7e471c70"
Sep 16 17:54:29.446678 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.446660    2581 scope.go:117] "RemoveContainer" containerID="96ea29857251e553a92ca3479d30b82000a7a8a6c8ee313825427ddc0fe4a135"
Sep 16 17:54:29.454875 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.454860    2581 scope.go:117] "RemoveContainer" containerID="eade65062087b035f8b2b8dcacee9acf1f32206126021723bdd57057c2d08222"
Sep 16 17:54:29.462386 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.462364    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"]
Sep 16 17:54:29.462961 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.462945    2581 scope.go:117] "RemoveContainer" containerID="208ca6fb27d22c138f7ee1c1640cde613a4c9eba90f3f4c5211b028b7e471c70"
Sep 16 17:54:29.463251 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:54:29.463233    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208ca6fb27d22c138f7ee1c1640cde613a4c9eba90f3f4c5211b028b7e471c70\": container with ID starting with 208ca6fb27d22c138f7ee1c1640cde613a4c9eba90f3f4c5211b028b7e471c70 not found: ID does not exist" containerID="208ca6fb27d22c138f7ee1c1640cde613a4c9eba90f3f4c5211b028b7e471c70"
Sep 16 17:54:29.463329 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.463263    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208ca6fb27d22c138f7ee1c1640cde613a4c9eba90f3f4c5211b028b7e471c70"} err="failed to get container status \"208ca6fb27d22c138f7ee1c1640cde613a4c9eba90f3f4c5211b028b7e471c70\": rpc error: code = NotFound desc = could not find container \"208ca6fb27d22c138f7ee1c1640cde613a4c9eba90f3f4c5211b028b7e471c70\": container with ID starting with 208ca6fb27d22c138f7ee1c1640cde613a4c9eba90f3f4c5211b028b7e471c70 not found: ID does not exist"
Sep 16 17:54:29.463329 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.463286    2581 scope.go:117] "RemoveContainer" containerID="96ea29857251e553a92ca3479d30b82000a7a8a6c8ee313825427ddc0fe4a135"
Sep 16 17:54:29.463569 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:54:29.463551    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ea29857251e553a92ca3479d30b82000a7a8a6c8ee313825427ddc0fe4a135\": container with ID starting with 96ea29857251e553a92ca3479d30b82000a7a8a6c8ee313825427ddc0fe4a135 not found: ID does not exist" containerID="96ea29857251e553a92ca3479d30b82000a7a8a6c8ee313825427ddc0fe4a135"
Sep 16 17:54:29.463611 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.463574    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ea29857251e553a92ca3479d30b82000a7a8a6c8ee313825427ddc0fe4a135"} err="failed to get container status \"96ea29857251e553a92ca3479d30b82000a7a8a6c8ee313825427ddc0fe4a135\": rpc error: code = NotFound desc = could not find container \"96ea29857251e553a92ca3479d30b82000a7a8a6c8ee313825427ddc0fe4a135\": container with ID starting with 96ea29857251e553a92ca3479d30b82000a7a8a6c8ee313825427ddc0fe4a135 not found: ID does not exist"
Sep 16 17:54:29.463611 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.463589    2581 scope.go:117] "RemoveContainer" containerID="eade65062087b035f8b2b8dcacee9acf1f32206126021723bdd57057c2d08222"
Sep 16 17:54:29.463836 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:54:29.463810    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eade65062087b035f8b2b8dcacee9acf1f32206126021723bdd57057c2d08222\": container with ID starting with eade65062087b035f8b2b8dcacee9acf1f32206126021723bdd57057c2d08222 not found: ID does not exist" containerID="eade65062087b035f8b2b8dcacee9acf1f32206126021723bdd57057c2d08222"
Sep 16 17:54:29.463906 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.463846    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eade65062087b035f8b2b8dcacee9acf1f32206126021723bdd57057c2d08222"} err="failed to get container status \"eade65062087b035f8b2b8dcacee9acf1f32206126021723bdd57057c2d08222\": rpc error: code = NotFound desc = could not find container \"eade65062087b035f8b2b8dcacee9acf1f32206126021723bdd57057c2d08222\": container with ID starting with eade65062087b035f8b2b8dcacee9acf1f32206126021723bdd57057c2d08222 not found: ID does not exist"
Sep 16 17:54:29.471480 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.471460    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljldd"]
Sep 16 17:54:29.766034 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:29.766011    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" path="/var/lib/kubelet/pods/f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09/volumes"
Sep 16 17:54:31.804233 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:31.804206    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:54:31.805690 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:31.805668    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:54:33.090101 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:33.090045    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:54:33.217693 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:33.217632    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:54:39.360645 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:39.360610    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw" podUID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 17:54:49.328516 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:49.328482    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:54:56.139436 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:56.139405    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"]
Sep 16 17:54:56.139928 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:56.139660    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw" podUID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerName="kserve-container" containerID="cri-o://5d4538466ef4f12e9338e3a0b2b7defff950f5a273ec2f6a10b17122f740b70c" gracePeriod=30
Sep 16 17:54:56.139928 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:56.139668    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw" podUID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerName="kube-rbac-proxy" containerID="cri-o://2d0d38374e2eb6dd4adc231649962f567d56000cc47718125729e551de63f867" gracePeriod=30
Sep 16 17:54:56.145804 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:56.145779    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:54:56.550374 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:56.550350    2581 generic.go:356] "Generic (PLEG): container finished" podID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerID="2d0d38374e2eb6dd4adc231649962f567d56000cc47718125729e551de63f867" exitCode=2
Sep 16 17:54:56.550522 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:56.550392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw" event={"ID":"8a6addb7-0378-4bf3-98b9-c98a665860c5","Type":"ContainerDied","Data":"2d0d38374e2eb6dd4adc231649962f567d56000cc47718125729e551de63f867"}
Sep 16 17:54:59.321590 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:59.321549    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw" podUID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 16 17:54:59.327044 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:54:59.327011    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw" podUID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.41:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 16 17:55:01.278493 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.278470    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:55:01.321069 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.321002    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a6addb7-0378-4bf3-98b9-c98a665860c5-kube-rbac-proxy-sar-config\") pod \"8a6addb7-0378-4bf3-98b9-c98a665860c5\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") "
Sep 16 17:55:01.321069 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.321056    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a6addb7-0378-4bf3-98b9-c98a665860c5-proxy-tls\") pod \"8a6addb7-0378-4bf3-98b9-c98a665860c5\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") "
Sep 16 17:55:01.321237 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.321083    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a6addb7-0378-4bf3-98b9-c98a665860c5-cabundle-cert\") pod \"8a6addb7-0378-4bf3-98b9-c98a665860c5\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") "
Sep 16 17:55:01.321237 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.321109    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-926xk\" (UniqueName: \"kubernetes.io/projected/8a6addb7-0378-4bf3-98b9-c98a665860c5-kube-api-access-926xk\") pod \"8a6addb7-0378-4bf3-98b9-c98a665860c5\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") "
Sep 16 17:55:01.321237 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.321140    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a6addb7-0378-4bf3-98b9-c98a665860c5-kserve-provision-location\") pod \"8a6addb7-0378-4bf3-98b9-c98a665860c5\" (UID: \"8a6addb7-0378-4bf3-98b9-c98a665860c5\") "
Sep 16 17:55:01.321552 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.321350    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6addb7-0378-4bf3-98b9-c98a665860c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a6addb7-0378-4bf3-98b9-c98a665860c5" (UID: "8a6addb7-0378-4bf3-98b9-c98a665860c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:55:01.321552 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.321528    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6addb7-0378-4bf3-98b9-c98a665860c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a6addb7-0378-4bf3-98b9-c98a665860c5" (UID: "8a6addb7-0378-4bf3-98b9-c98a665860c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:55:01.322071 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.321664    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6addb7-0378-4bf3-98b9-c98a665860c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a6addb7-0378-4bf3-98b9-c98a665860c5" (UID: "8a6addb7-0378-4bf3-98b9-c98a665860c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:55:01.323452 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.323433    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6addb7-0378-4bf3-98b9-c98a665860c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a6addb7-0378-4bf3-98b9-c98a665860c5" (UID: "8a6addb7-0378-4bf3-98b9-c98a665860c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:55:01.324369 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.324341    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6addb7-0378-4bf3-98b9-c98a665860c5-kube-api-access-926xk" (OuterVolumeSpecName: "kube-api-access-926xk") pod "8a6addb7-0378-4bf3-98b9-c98a665860c5" (UID: "8a6addb7-0378-4bf3-98b9-c98a665860c5"). InnerVolumeSpecName "kube-api-access-926xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:55:01.421656 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.421625    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a6addb7-0378-4bf3-98b9-c98a665860c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:55:01.421656 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.421647    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a6addb7-0378-4bf3-98b9-c98a665860c5-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:55:01.421656 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.421659    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a6addb7-0378-4bf3-98b9-c98a665860c5-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:55:01.421886 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.421668    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-926xk\" (UniqueName: \"kubernetes.io/projected/8a6addb7-0378-4bf3-98b9-c98a665860c5-kube-api-access-926xk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:55:01.421886 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.421676    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a6addb7-0378-4bf3-98b9-c98a665860c5-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:55:01.450854 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.450829    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:01.457708 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.457690    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"]
Sep 16 17:55:01.458015 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458002    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" containerName="kube-rbac-proxy"
Sep 16 17:55:01.458084 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458017    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" containerName="kube-rbac-proxy"
Sep 16 17:55:01.458084 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458029    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerName="storage-initializer"
Sep 16 17:55:01.458084 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458036    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerName="storage-initializer"
Sep 16 17:55:01.458084 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458048    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerName="kserve-container"
Sep 16 17:55:01.458084 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458053    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerName="kserve-container"
Sep 16 17:55:01.458084 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458061    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" containerName="storage-initializer"
Sep 16 17:55:01.458084 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458066    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" containerName="storage-initializer"
Sep 16 17:55:01.458084 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458077    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" containerName="kserve-container"
Sep 16 17:55:01.458084 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458082    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" containerName="kserve-container"
Sep 16 17:55:01.458084 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458088    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerName="kube-rbac-proxy"
Sep 16 17:55:01.458552 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458093    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerName="kube-rbac-proxy"
Sep 16 17:55:01.458552 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458141    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" containerName="kserve-container"
Sep 16 17:55:01.458552 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458148    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerName="kube-rbac-proxy"
Sep 16 17:55:01.458552 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458156    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ba77ba-634e-4f79-8ab3-4d3a2d3a0d09" containerName="kube-rbac-proxy"
Sep 16 17:55:01.458552 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.458161    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerName="kserve-container"
Sep 16 17:55:01.461348 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.461333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:01.465171 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.465154    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 16 17:55:01.465967 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.465952    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 16 17:55:01.488183 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.488161    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"]
Sep 16 17:55:01.522580 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.522561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:01.522667 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.522593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:01.522667 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.522614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-proxy-tls\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:01.522776 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.522735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:01.522807 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.522786    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4djj\" (UniqueName: \"kubernetes.io/projected/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kube-api-access-s4djj\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:01.579623 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.579564    2581 generic.go:356] "Generic (PLEG): container finished" podID="8a6addb7-0378-4bf3-98b9-c98a665860c5" containerID="5d4538466ef4f12e9338e3a0b2b7defff950f5a273ec2f6a10b17122f740b70c" exitCode=0
Sep 16 17:55:01.579623 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.579597    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw" event={"ID":"8a6addb7-0378-4bf3-98b9-c98a665860c5","Type":"ContainerDied","Data":"5d4538466ef4f12e9338e3a0b2b7defff950f5a273ec2f6a10b17122f740b70c"}
Sep 16 17:55:01.579623 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.579619    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"
Sep 16 17:55:01.579815 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.579647    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw" event={"ID":"8a6addb7-0378-4bf3-98b9-c98a665860c5","Type":"ContainerDied","Data":"15ace0c3083e0e857b7d6a1fc21a974c5272b7efc332f9659d7b604424412447"}
Sep 16 17:55:01.579815 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.579684    2581 scope.go:117] "RemoveContainer" containerID="2d0d38374e2eb6dd4adc231649962f567d56000cc47718125729e551de63f867"
Sep 16 17:55:01.588294 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.588277    2581 scope.go:117] "RemoveContainer" containerID="5d4538466ef4f12e9338e3a0b2b7defff950f5a273ec2f6a10b17122f740b70c"
Sep 16 17:55:01.596780 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.596753    2581 scope.go:117] "RemoveContainer" containerID="caef26be73014f08f885a03f3dc3afaba97d55f87939181912197883c94f5a9e"
Sep 16 17:55:01.603809 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.603790    2581 scope.go:117] "RemoveContainer" containerID="2d0d38374e2eb6dd4adc231649962f567d56000cc47718125729e551de63f867"
Sep 16 17:55:01.604112 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:55:01.604090    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0d38374e2eb6dd4adc231649962f567d56000cc47718125729e551de63f867\": container with ID starting with 2d0d38374e2eb6dd4adc231649962f567d56000cc47718125729e551de63f867 not found: ID does not exist" containerID="2d0d38374e2eb6dd4adc231649962f567d56000cc47718125729e551de63f867"
Sep 16 17:55:01.604180 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.604123    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0d38374e2eb6dd4adc231649962f567d56000cc47718125729e551de63f867"} err="failed to get container status \"2d0d38374e2eb6dd4adc231649962f567d56000cc47718125729e551de63f867\": rpc error: code = NotFound desc = could not find container \"2d0d38374e2eb6dd4adc231649962f567d56000cc47718125729e551de63f867\": container with ID starting with 2d0d38374e2eb6dd4adc231649962f567d56000cc47718125729e551de63f867 not found: ID does not exist"
Sep 16 17:55:01.604180 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.604146    2581 scope.go:117] "RemoveContainer" containerID="5d4538466ef4f12e9338e3a0b2b7defff950f5a273ec2f6a10b17122f740b70c"
Sep 16 17:55:01.604421 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:55:01.604404    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4538466ef4f12e9338e3a0b2b7defff950f5a273ec2f6a10b17122f740b70c\": container with ID starting with 5d4538466ef4f12e9338e3a0b2b7defff950f5a273ec2f6a10b17122f740b70c not found: ID does not exist" containerID="5d4538466ef4f12e9338e3a0b2b7defff950f5a273ec2f6a10b17122f740b70c"
Sep 16 17:55:01.604476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.604427    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4538466ef4f12e9338e3a0b2b7defff950f5a273ec2f6a10b17122f740b70c"} err="failed to get container status \"5d4538466ef4f12e9338e3a0b2b7defff950f5a273ec2f6a10b17122f740b70c\": rpc error: code = NotFound desc = could not find container \"5d4538466ef4f12e9338e3a0b2b7defff950f5a273ec2f6a10b17122f740b70c\": container with ID starting with 5d4538466ef4f12e9338e3a0b2b7defff950f5a273ec2f6a10b17122f740b70c not found: ID does not exist"
Sep 16 17:55:01.604476 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.604444    2581 scope.go:117] "RemoveContainer" containerID="caef26be73014f08f885a03f3dc3afaba97d55f87939181912197883c94f5a9e"
Sep 16 17:55:01.604688 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:55:01.604674    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caef26be73014f08f885a03f3dc3afaba97d55f87939181912197883c94f5a9e\": container with ID starting with caef26be73014f08f885a03f3dc3afaba97d55f87939181912197883c94f5a9e not found: ID does not exist" containerID="caef26be73014f08f885a03f3dc3afaba97d55f87939181912197883c94f5a9e"
Sep 16 17:55:01.604741 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.604689    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caef26be73014f08f885a03f3dc3afaba97d55f87939181912197883c94f5a9e"} err="failed to get container status \"caef26be73014f08f885a03f3dc3afaba97d55f87939181912197883c94f5a9e\": rpc error: code = NotFound desc = could not find container \"caef26be73014f08f885a03f3dc3afaba97d55f87939181912197883c94f5a9e\": container with ID starting with caef26be73014f08f885a03f3dc3afaba97d55f87939181912197883c94f5a9e not found: ID does not exist"
Sep 16 17:55:01.614392 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.614371    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"]
Sep 16 17:55:01.622563 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.622545    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-h6zdw"]
Sep 16 17:55:01.623447 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.623416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:01.623543 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.623464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4djj\" (UniqueName: \"kubernetes.io/projected/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kube-api-access-s4djj\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:01.623543 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.623518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:01.623654 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.623548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:01.623654 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.623579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-proxy-tls\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:01.623756 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:55:01.623729    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 16 17:55:01.623756 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.623738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:01.623852 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:55:01.623783    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-proxy-tls podName:feaa5ffc-65d1-4ce0-88d1-f3201f90e977 nodeName:}" failed. No retries permitted until 2026-09-16 17:55:02.123764926 +0000 UTC m=+1830.912387451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-proxy-tls") pod "isvc-sklearn-v2-predictor-64bfb47598-p4xbh" (UID: "feaa5ffc-65d1-4ce0-88d1-f3201f90e977") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 16 17:55:01.624190 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.624173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:01.624230 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.624176    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:01.634399 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.634382    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4djj\" (UniqueName: \"kubernetes.io/projected/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kube-api-access-s4djj\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:01.765723 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.765689    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a6addb7-0378-4bf3-98b9-c98a665860c5" path="/var/lib/kubelet/pods/8a6addb7-0378-4bf3-98b9-c98a665860c5/volumes"
Sep 16 17:55:01.940850 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:01.940797    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:02.126340 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:02.126317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-proxy-tls\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:02.128646 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:02.128627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-proxy-tls\") pod \"isvc-sklearn-v2-predictor-64bfb47598-p4xbh\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:02.370481 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:02.370451    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:02.646234 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:02.646212    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"]
Sep 16 17:55:02.648189 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:55:02.648164    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeaa5ffc_65d1_4ce0_88d1_f3201f90e977.slice/crio-f072f3d4914c79b82f10ff7c80afa3eb0c179eeae5d1bccdbfe4a5cafbd2c5a2 WatchSource:0}: Error finding container f072f3d4914c79b82f10ff7c80afa3eb0c179eeae5d1bccdbfe4a5cafbd2c5a2: Status 404 returned error can't find the container with id f072f3d4914c79b82f10ff7c80afa3eb0c179eeae5d1bccdbfe4a5cafbd2c5a2
Sep 16 17:55:03.595908 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:03.595868    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" event={"ID":"feaa5ffc-65d1-4ce0-88d1-f3201f90e977","Type":"ContainerStarted","Data":"8040f8bb9c45493371a5e59117b48469da01d6c03383e7e40a32c0a5cae6064f"}
Sep 16 17:55:03.596345 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:03.595934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" event={"ID":"feaa5ffc-65d1-4ce0-88d1-f3201f90e977","Type":"ContainerStarted","Data":"f072f3d4914c79b82f10ff7c80afa3eb0c179eeae5d1bccdbfe4a5cafbd2c5a2"}
Sep 16 17:55:06.609539 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:06.609507    2581 generic.go:356] "Generic (PLEG): container finished" podID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerID="8040f8bb9c45493371a5e59117b48469da01d6c03383e7e40a32c0a5cae6064f" exitCode=0
Sep 16 17:55:06.609824 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:06.609576    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" event={"ID":"feaa5ffc-65d1-4ce0-88d1-f3201f90e977","Type":"ContainerDied","Data":"8040f8bb9c45493371a5e59117b48469da01d6c03383e7e40a32c0a5cae6064f"}
Sep 16 17:55:07.616232 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:07.616200    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" event={"ID":"feaa5ffc-65d1-4ce0-88d1-f3201f90e977","Type":"ContainerStarted","Data":"2799d3759fcb7c7ee162db60484dd15d7f4659041a0d9106d384df7a22bdfc12"}
Sep 16 17:55:07.616585 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:07.616255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" event={"ID":"feaa5ffc-65d1-4ce0-88d1-f3201f90e977","Type":"ContainerStarted","Data":"5015c313587d2630a27b8b91dd63a62c5059657618e99f1c217a86882a30062a"}
Sep 16 17:55:07.616585 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:07.616427    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:07.639192 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:07.639148    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" podStartSLOduration=2.773705419 podStartE2EDuration="6.639136334s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.865430915s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:55:07.637593804 +0000 UTC m=+1836.426216359" watchObservedRunningTime="2026-09-16 17:55:07.639136334 +0000 UTC m=+1836.427758867"
Sep 16 17:55:08.621410 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:08.621372    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:08.622728 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:08.622700    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 17:55:09.625660 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:09.625628    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 17:55:14.630111 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:14.630084    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:55:14.630605 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:14.630578    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 17:55:19.304134 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:19.304096    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:24.379217 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:24.379183    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:24.630694 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:24.630621    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 17:55:25.146904 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:25.146866    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:34.631434 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:34.631398    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 17:55:44.631480 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:44.631445    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 17:55:54.630753 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:54.630668    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 17:55:59.579378 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:55:59.579340    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:56:04.630725 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.630688    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 17:56:04.737055 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.737019    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:56:04.839014 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.838984    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"]
Sep 16 17:56:04.842466 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.842447    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:04.849927 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.849882    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 16 17:56:04.850555 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.850518    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 17:56:04.850845 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.850820    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:56:04.882920 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.882851    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"]
Sep 16 17:56:04.931593 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.931569    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"]
Sep 16 17:56:04.935191 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.935175    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:04.937948 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.937932    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 17:56:04.959049 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.959023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s7kvs\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:04.959194 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.959054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s7kvs\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:04.959194 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.959100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s7kvs\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:04.959194 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.959164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xglkb\" (UniqueName: \"kubernetes.io/projected/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kube-api-access-xglkb\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s7kvs\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:04.959381 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.959195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s7kvs\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:04.962137 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.962117    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"]
Sep 16 17:56:04.965813 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:04.965786    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:56:05.060318 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.060295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s7kvs\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:05.060414 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.060337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.060414 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.060355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s7kvs\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:05.060528 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.060449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.060528 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.060480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xglkb\" (UniqueName: \"kubernetes.io/projected/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kube-api-access-xglkb\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s7kvs\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:05.060528 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.060498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s7kvs\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:05.060528 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.060516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.060736 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.060540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.060736 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.060590    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvcd6\" (UniqueName: \"kubernetes.io/projected/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kube-api-access-bvcd6\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.060736 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.060622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s7kvs\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:05.060736 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.060686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s7kvs\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:05.061142 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.061123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s7kvs\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:05.061284 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.061265    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s7kvs\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:05.063202 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.063174    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s7kvs\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:05.086262 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.086243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xglkb\" (UniqueName: \"kubernetes.io/projected/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kube-api-access-xglkb\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-s7kvs\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:05.154453 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.154408    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:05.163388 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.163364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvcd6\" (UniqueName: \"kubernetes.io/projected/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kube-api-access-bvcd6\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.163501 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.163455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.163501 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.163487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.163612 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.163532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.163612 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.163560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.163872 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:56:05.163851    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 16 17:56:05.164106 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:56:05.164037    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-proxy-tls podName:a8723ca3-5bc1-4199-88f3-edd9b413a3a7 nodeName:}" failed. No retries permitted until 2026-09-16 17:56:05.66401563 +0000 UTC m=+1894.452638161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-proxy-tls") pod "isvc-canary-create-predictor-854596548c-4rtq5" (UID: "a8723ca3-5bc1-4199-88f3-edd9b413a3a7") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 16 17:56:05.164106 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.163954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.164367 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.164348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.164519 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.164496    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.184992 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.184968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvcd6\" (UniqueName: \"kubernetes.io/projected/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kube-api-access-bvcd6\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.326841 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.326808    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:56:05.441673 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.441610    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"]
Sep 16 17:56:05.444702 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:56:05.444677    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4c765c_d52d_4ab5_ae19_10afbc9f6a8a.slice/crio-bad93dc950535b251270adc3d9a3d383948db7e02a119c2174b39cd2b850d300 WatchSource:0}: Error finding container bad93dc950535b251270adc3d9a3d383948db7e02a119c2174b39cd2b850d300: Status 404 returned error can't find the container with id bad93dc950535b251270adc3d9a3d383948db7e02a119c2174b39cd2b850d300
Sep 16 17:56:05.668320 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.668290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.670628 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.670607    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-4rtq5\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.843982 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.843951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:05.852322 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.852294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" event={"ID":"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a","Type":"ContainerStarted","Data":"85e17cd370ef9b49d6a94e103ba0ff3a0efb5fc51a553e31b6ba9e4f4eb92b77"}
Sep 16 17:56:05.852435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:05.852351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" event={"ID":"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a","Type":"ContainerStarted","Data":"bad93dc950535b251270adc3d9a3d383948db7e02a119c2174b39cd2b850d300"}
Sep 16 17:56:06.132308 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:06.132175    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"]
Sep 16 17:56:06.134261 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:56:06.134227    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8723ca3_5bc1_4199_88f3_edd9b413a3a7.slice/crio-020311c8f0c755da5f6ffb44665ff9ed17e0787ad56c0e6970fe3b41bd25036d WatchSource:0}: Error finding container 020311c8f0c755da5f6ffb44665ff9ed17e0787ad56c0e6970fe3b41bd25036d: Status 404 returned error can't find the container with id 020311c8f0c755da5f6ffb44665ff9ed17e0787ad56c0e6970fe3b41bd25036d
Sep 16 17:56:06.858458 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:06.858424    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" event={"ID":"a8723ca3-5bc1-4199-88f3-edd9b413a3a7","Type":"ContainerStarted","Data":"c26c8edc73d33238a25f90dbaad2dfb5302cd8c61ad64dec560ef8d689e62231"}
Sep 16 17:56:06.858837 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:06.858478    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" event={"ID":"a8723ca3-5bc1-4199-88f3-edd9b413a3a7","Type":"ContainerStarted","Data":"020311c8f0c755da5f6ffb44665ff9ed17e0787ad56c0e6970fe3b41bd25036d"}
Sep 16 17:56:09.872862 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:09.872837    2581 generic.go:356] "Generic (PLEG): container finished" podID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerID="85e17cd370ef9b49d6a94e103ba0ff3a0efb5fc51a553e31b6ba9e4f4eb92b77" exitCode=0
Sep 16 17:56:09.873228 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:09.872881    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" event={"ID":"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a","Type":"ContainerDied","Data":"85e17cd370ef9b49d6a94e103ba0ff3a0efb5fc51a553e31b6ba9e4f4eb92b77"}
Sep 16 17:56:10.878388 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:10.878355    2581 generic.go:356] "Generic (PLEG): container finished" podID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerID="c26c8edc73d33238a25f90dbaad2dfb5302cd8c61ad64dec560ef8d689e62231" exitCode=0
Sep 16 17:56:10.878840 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:10.878432    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" event={"ID":"a8723ca3-5bc1-4199-88f3-edd9b413a3a7","Type":"ContainerDied","Data":"c26c8edc73d33238a25f90dbaad2dfb5302cd8c61ad64dec560ef8d689e62231"}
Sep 16 17:56:10.880770 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:10.880750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" event={"ID":"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a","Type":"ContainerStarted","Data":"4a84943deecedc43c1d136497b23f345d60b52ae2678bb1429695b204766519b"}
Sep 16 17:56:10.880863 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:10.880798    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" event={"ID":"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a","Type":"ContainerStarted","Data":"c17bc7550b4d8bea062c3909856a34d61c49d70497fa1024a4e5fae3f2479711"}
Sep 16 17:56:10.881168 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:10.881071    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:10.881168 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:10.881127    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:10.882243 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:10.882220    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 17:56:10.941377 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:10.941310    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" podStartSLOduration=3.032296298 podStartE2EDuration="6.941296193s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.908999895s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:56:10.938325494 +0000 UTC m=+1899.726948027" watchObservedRunningTime="2026-09-16 17:56:10.941296193 +0000 UTC m=+1899.729918725"
Sep 16 17:56:11.887031 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:11.886998    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" event={"ID":"a8723ca3-5bc1-4199-88f3-edd9b413a3a7","Type":"ContainerStarted","Data":"17f6aea38fcf71fd9aa1159b1d9a76a2be06b9db901832ccd95a29d301d12a4b"}
Sep 16 17:56:11.887422 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:11.887054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" event={"ID":"a8723ca3-5bc1-4199-88f3-edd9b413a3a7","Type":"ContainerStarted","Data":"0dee1ed8439793f9397749643c258971fd3684f1722a51d1322f2fbdc727057d"}
Sep 16 17:56:11.887422 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:11.887236    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 17:56:11.887422 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:11.887390    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:11.887584 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:11.887524    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:11.888781 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:11.888750    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 17:56:11.921313 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:11.921270    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" podStartSLOduration=3.906588031 podStartE2EDuration="7.921259739s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.014671708s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:56:11.918764994 +0000 UTC m=+1900.707387526" watchObservedRunningTime="2026-09-16 17:56:11.921259739 +0000 UTC m=+1900.709882271"
Sep 16 17:56:12.891610 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:12.891569    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 17:56:14.631906 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:14.631865    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:56:16.891540 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:16.891509    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:56:16.892108 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:16.892079    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 17:56:17.896638 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:17.896610    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:56:17.897047 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:17.897026    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 17:56:21.625827 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:21.625791    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"]
Sep 16 17:56:21.626249 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:21.626027    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kserve-container" containerID="cri-o://5015c313587d2630a27b8b91dd63a62c5059657618e99f1c217a86882a30062a" gracePeriod=30
Sep 16 17:56:21.626249 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:21.626066    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kube-rbac-proxy" containerID="cri-o://2799d3759fcb7c7ee162db60484dd15d7f4659041a0d9106d384df7a22bdfc12" gracePeriod=30
Sep 16 17:56:21.664959 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:21.664927    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:56:21.931840 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:21.931758    2581 generic.go:356] "Generic (PLEG): container finished" podID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerID="2799d3759fcb7c7ee162db60484dd15d7f4659041a0d9106d384df7a22bdfc12" exitCode=2
Sep 16 17:56:21.931987 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:21.931835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" event={"ID":"feaa5ffc-65d1-4ce0-88d1-f3201f90e977","Type":"ContainerDied","Data":"2799d3759fcb7c7ee162db60484dd15d7f4659041a0d9106d384df7a22bdfc12"}
Sep 16 17:56:24.625885 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:24.625846    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 16 17:56:24.631180 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:24.631159    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 16 17:56:25.570816 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.570790    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:56:25.715325 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.715260    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-proxy-tls\") pod \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") "
Sep 16 17:56:25.715325 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.715313    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4djj\" (UniqueName: \"kubernetes.io/projected/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kube-api-access-s4djj\") pod \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") "
Sep 16 17:56:25.715735 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.715344    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kube-rbac-proxy-sar-config\") pod \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") "
Sep 16 17:56:25.715735 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.715378    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-cabundle-cert\") pod \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") "
Sep 16 17:56:25.715735 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.715405    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kserve-provision-location\") pod \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\" (UID: \"feaa5ffc-65d1-4ce0-88d1-f3201f90e977\") "
Sep 16 17:56:25.715735 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.715710    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "feaa5ffc-65d1-4ce0-88d1-f3201f90e977" (UID: "feaa5ffc-65d1-4ce0-88d1-f3201f90e977"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:56:25.715948 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.715727    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "feaa5ffc-65d1-4ce0-88d1-f3201f90e977" (UID: "feaa5ffc-65d1-4ce0-88d1-f3201f90e977"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:56:25.715948 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.715770    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "feaa5ffc-65d1-4ce0-88d1-f3201f90e977" (UID: "feaa5ffc-65d1-4ce0-88d1-f3201f90e977"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:56:25.717562 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.717545    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kube-api-access-s4djj" (OuterVolumeSpecName: "kube-api-access-s4djj") pod "feaa5ffc-65d1-4ce0-88d1-f3201f90e977" (UID: "feaa5ffc-65d1-4ce0-88d1-f3201f90e977"). InnerVolumeSpecName "kube-api-access-s4djj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:56:25.717828 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.717815    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "feaa5ffc-65d1-4ce0-88d1-f3201f90e977" (UID: "feaa5ffc-65d1-4ce0-88d1-f3201f90e977"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:56:25.816418 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.816393    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:56:25.816418 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.816415    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4djj\" (UniqueName: \"kubernetes.io/projected/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kube-api-access-s4djj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:56:25.816548 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.816424    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:56:25.816548 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.816433    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:56:25.816548 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.816442    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feaa5ffc-65d1-4ce0-88d1-f3201f90e977-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:56:25.951109 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.951084    2581 generic.go:356] "Generic (PLEG): container finished" podID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerID="5015c313587d2630a27b8b91dd63a62c5059657618e99f1c217a86882a30062a" exitCode=0
Sep 16 17:56:25.951222 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.951139    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"
Sep 16 17:56:25.951222 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.951163    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" event={"ID":"feaa5ffc-65d1-4ce0-88d1-f3201f90e977","Type":"ContainerDied","Data":"5015c313587d2630a27b8b91dd63a62c5059657618e99f1c217a86882a30062a"}
Sep 16 17:56:25.951222 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.951220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh" event={"ID":"feaa5ffc-65d1-4ce0-88d1-f3201f90e977","Type":"ContainerDied","Data":"f072f3d4914c79b82f10ff7c80afa3eb0c179eeae5d1bccdbfe4a5cafbd2c5a2"}
Sep 16 17:56:25.951325 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.951247    2581 scope.go:117] "RemoveContainer" containerID="2799d3759fcb7c7ee162db60484dd15d7f4659041a0d9106d384df7a22bdfc12"
Sep 16 17:56:25.961911 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.961663    2581 scope.go:117] "RemoveContainer" containerID="5015c313587d2630a27b8b91dd63a62c5059657618e99f1c217a86882a30062a"
Sep 16 17:56:25.970932 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.970914    2581 scope.go:117] "RemoveContainer" containerID="8040f8bb9c45493371a5e59117b48469da01d6c03383e7e40a32c0a5cae6064f"
Sep 16 17:56:25.979160 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.979088    2581 scope.go:117] "RemoveContainer" containerID="2799d3759fcb7c7ee162db60484dd15d7f4659041a0d9106d384df7a22bdfc12"
Sep 16 17:56:25.979382 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:56:25.979361    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2799d3759fcb7c7ee162db60484dd15d7f4659041a0d9106d384df7a22bdfc12\": container with ID starting with 2799d3759fcb7c7ee162db60484dd15d7f4659041a0d9106d384df7a22bdfc12 not found: ID does not exist" containerID="2799d3759fcb7c7ee162db60484dd15d7f4659041a0d9106d384df7a22bdfc12"
Sep 16 17:56:25.979465 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.979393    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2799d3759fcb7c7ee162db60484dd15d7f4659041a0d9106d384df7a22bdfc12"} err="failed to get container status \"2799d3759fcb7c7ee162db60484dd15d7f4659041a0d9106d384df7a22bdfc12\": rpc error: code = NotFound desc = could not find container \"2799d3759fcb7c7ee162db60484dd15d7f4659041a0d9106d384df7a22bdfc12\": container with ID starting with 2799d3759fcb7c7ee162db60484dd15d7f4659041a0d9106d384df7a22bdfc12 not found: ID does not exist"
Sep 16 17:56:25.979465 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.979414    2581 scope.go:117] "RemoveContainer" containerID="5015c313587d2630a27b8b91dd63a62c5059657618e99f1c217a86882a30062a"
Sep 16 17:56:25.979675 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:56:25.979658    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5015c313587d2630a27b8b91dd63a62c5059657618e99f1c217a86882a30062a\": container with ID starting with 5015c313587d2630a27b8b91dd63a62c5059657618e99f1c217a86882a30062a not found: ID does not exist" containerID="5015c313587d2630a27b8b91dd63a62c5059657618e99f1c217a86882a30062a"
Sep 16 17:56:25.979720 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.979681    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5015c313587d2630a27b8b91dd63a62c5059657618e99f1c217a86882a30062a"} err="failed to get container status \"5015c313587d2630a27b8b91dd63a62c5059657618e99f1c217a86882a30062a\": rpc error: code = NotFound desc = could not find container \"5015c313587d2630a27b8b91dd63a62c5059657618e99f1c217a86882a30062a\": container with ID starting with 5015c313587d2630a27b8b91dd63a62c5059657618e99f1c217a86882a30062a not found: ID does not exist"
Sep 16 17:56:25.979720 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.979698    2581 scope.go:117] "RemoveContainer" containerID="8040f8bb9c45493371a5e59117b48469da01d6c03383e7e40a32c0a5cae6064f"
Sep 16 17:56:25.979975 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:56:25.979958    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8040f8bb9c45493371a5e59117b48469da01d6c03383e7e40a32c0a5cae6064f\": container with ID starting with 8040f8bb9c45493371a5e59117b48469da01d6c03383e7e40a32c0a5cae6064f not found: ID does not exist" containerID="8040f8bb9c45493371a5e59117b48469da01d6c03383e7e40a32c0a5cae6064f"
Sep 16 17:56:25.980036 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.979980    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8040f8bb9c45493371a5e59117b48469da01d6c03383e7e40a32c0a5cae6064f"} err="failed to get container status \"8040f8bb9c45493371a5e59117b48469da01d6c03383e7e40a32c0a5cae6064f\": rpc error: code = NotFound desc = could not find container \"8040f8bb9c45493371a5e59117b48469da01d6c03383e7e40a32c0a5cae6064f\": container with ID starting with 8040f8bb9c45493371a5e59117b48469da01d6c03383e7e40a32c0a5cae6064f not found: ID does not exist"
Sep 16 17:56:25.980779 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.980763    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"]
Sep 16 17:56:25.988829 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:25.988807    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-p4xbh"]
Sep 16 17:56:26.794573 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:26.794547    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:56:26.892300 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:26.892272    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 17:56:26.910831 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:26.910803    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"]
Sep 16 17:56:26.911315 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:26.911300    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="storage-initializer"
Sep 16 17:56:26.911361 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:26.911319    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="storage-initializer"
Sep 16 17:56:26.911361 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:26.911334    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kube-rbac-proxy"
Sep 16 17:56:26.911361 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:26.911344    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kube-rbac-proxy"
Sep 16 17:56:26.911451 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:26.911361    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kserve-container"
Sep 16 17:56:26.911451 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:26.911370    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kserve-container"
Sep 16 17:56:26.911515 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:26.911459    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kube-rbac-proxy"
Sep 16 17:56:26.911515 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:26.911469    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" containerName="kserve-container"
Sep 16 17:56:26.915946 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:26.915928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:26.920201 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:26.920184    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 16 17:56:26.920481 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:26.920467    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 16 17:56:26.940849 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:26.940819    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"]
Sep 16 17:56:27.025440 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.025413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znsvs\" (UniqueName: \"kubernetes.io/projected/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kube-api-access-znsvs\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.025583 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.025445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5248c29d-bb8a-4a81-8984-4fbecd036a2d-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.025583 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.025471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.025583 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.025487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.025583 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.025555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5248c29d-bb8a-4a81-8984-4fbecd036a2d-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.119303 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.119241    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:56:27.126285 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.126268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.126348 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.126294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.126348 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.126314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5248c29d-bb8a-4a81-8984-4fbecd036a2d-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.126450 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.126358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znsvs\" (UniqueName: \"kubernetes.io/projected/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kube-api-access-znsvs\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.126497 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.126454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5248c29d-bb8a-4a81-8984-4fbecd036a2d-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.126668 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.126648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.126965 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.126944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.127049 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.126990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5248c29d-bb8a-4a81-8984-4fbecd036a2d-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.128869 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.128850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5248c29d-bb8a-4a81-8984-4fbecd036a2d-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.139503 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.139483    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znsvs\" (UniqueName: \"kubernetes.io/projected/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kube-api-access-znsvs\") pod \"isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.225977 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.225956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:27.517562 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.517540    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"]
Sep 16 17:56:27.519873 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:56:27.519845    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5248c29d_bb8a_4a81_8984_4fbecd036a2d.slice/crio-0b6c1a8d53f01ba43669b3f9e1c0a5e9e6b018669f91755ee4d86f404045f24d WatchSource:0}: Error finding container 0b6c1a8d53f01ba43669b3f9e1c0a5e9e6b018669f91755ee4d86f404045f24d: Status 404 returned error can't find the container with id 0b6c1a8d53f01ba43669b3f9e1c0a5e9e6b018669f91755ee4d86f404045f24d
Sep 16 17:56:27.766637 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.766609    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feaa5ffc-65d1-4ce0-88d1-f3201f90e977" path="/var/lib/kubelet/pods/feaa5ffc-65d1-4ce0-88d1-f3201f90e977/volumes"
Sep 16 17:56:27.897799 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.897760    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 17:56:27.962615 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.962581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" event={"ID":"5248c29d-bb8a-4a81-8984-4fbecd036a2d","Type":"ContainerStarted","Data":"05b994996bb6cb552149db5902a223c9d789b4171e8d6762cc73a37ec809751b"}
Sep 16 17:56:27.962794 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:27.962642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" event={"ID":"5248c29d-bb8a-4a81-8984-4fbecd036a2d","Type":"ContainerStarted","Data":"0b6c1a8d53f01ba43669b3f9e1c0a5e9e6b018669f91755ee4d86f404045f24d"}
Sep 16 17:56:31.984382 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:31.984354    2581 generic.go:356] "Generic (PLEG): container finished" podID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerID="05b994996bb6cb552149db5902a223c9d789b4171e8d6762cc73a37ec809751b" exitCode=0
Sep 16 17:56:31.984727 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:31.984401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" event={"ID":"5248c29d-bb8a-4a81-8984-4fbecd036a2d","Type":"ContainerDied","Data":"05b994996bb6cb552149db5902a223c9d789b4171e8d6762cc73a37ec809751b"}
Sep 16 17:56:32.993986 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:32.993939    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" event={"ID":"5248c29d-bb8a-4a81-8984-4fbecd036a2d","Type":"ContainerStarted","Data":"121d2aacc6953a2d10137e6c0274256f681d2604b91389beb3188894a62df8ea"}
Sep 16 17:56:32.994404 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:32.994008    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" event={"ID":"5248c29d-bb8a-4a81-8984-4fbecd036a2d","Type":"ContainerStarted","Data":"949b5a7a132a0613da503988d4a0729ba4689b941fb0b95d2419b9e6df4dcfe9"}
Sep 16 17:56:32.994404 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:32.994299    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:32.994524 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:32.994448    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:32.995498 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:32.995475    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 17:56:33.036757 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:33.036717    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" podStartSLOduration=3.156174207 podStartE2EDuration="7.036704312s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.880530105s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:56:33.034474895 +0000 UTC m=+1921.823097438" watchObservedRunningTime="2026-09-16 17:56:33.036704312 +0000 UTC m=+1921.825326844"
Sep 16 17:56:33.999091 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:33.999051    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 17:56:36.892173 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:36.892094    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 17:56:37.897319 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:37.897282    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 17:56:39.002672 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:39.002645    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:56:39.003137 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:39.003112    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 17:56:46.892165 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:46.892127    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 17:56:47.897043 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:47.897007    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 17:56:49.003162 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:49.003111    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 17:56:56.892297 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:56.892260    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 17:56:57.897698 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:57.897655    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 17:56:59.003063 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:56:59.003027    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 17:57:06.892469 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:06.892420    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 17:57:07.897384 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:07.897352    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 17:57:09.003642 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:09.003607    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 17:57:13.904923 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:13.904875    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:16.892460 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:16.892420    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 17:57:17.897123 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:17.897086    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 17:57:19.003404 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:19.003365    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 17:57:26.893566 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:26.893533    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:57:27.257852 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:27.257770    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"]
Sep 16 17:57:27.258086 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:27.258055    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kserve-container" containerID="cri-o://0dee1ed8439793f9397749643c258971fd3684f1722a51d1322f2fbdc727057d" gracePeriod=30
Sep 16 17:57:27.258203 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:27.258119    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kube-rbac-proxy" containerID="cri-o://17f6aea38fcf71fd9aa1159b1d9a76a2be06b9db901832ccd95a29d301d12a4b" gracePeriod=30
Sep 16 17:57:27.258931 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:27.258914    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:57:27.892097 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:27.892067    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 16 17:57:28.244418 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:28.244336    2581 generic.go:356] "Generic (PLEG): container finished" podID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerID="17f6aea38fcf71fd9aa1159b1d9a76a2be06b9db901832ccd95a29d301d12a4b" exitCode=2
Sep 16 17:57:28.244418 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:28.244386    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" event={"ID":"a8723ca3-5bc1-4199-88f3-edd9b413a3a7","Type":"ContainerDied","Data":"17f6aea38fcf71fd9aa1159b1d9a76a2be06b9db901832ccd95a29d301d12a4b"}
Sep 16 17:57:29.003106 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:29.003072    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 17:57:31.197928 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.197886    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:57:31.262390 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.262326    2581 generic.go:356] "Generic (PLEG): container finished" podID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerID="0dee1ed8439793f9397749643c258971fd3684f1722a51d1322f2fbdc727057d" exitCode=0
Sep 16 17:57:31.262390 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.262385    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"
Sep 16 17:57:31.262569 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.262414    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" event={"ID":"a8723ca3-5bc1-4199-88f3-edd9b413a3a7","Type":"ContainerDied","Data":"0dee1ed8439793f9397749643c258971fd3684f1722a51d1322f2fbdc727057d"}
Sep 16 17:57:31.262569 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.262486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5" event={"ID":"a8723ca3-5bc1-4199-88f3-edd9b413a3a7","Type":"ContainerDied","Data":"020311c8f0c755da5f6ffb44665ff9ed17e0787ad56c0e6970fe3b41bd25036d"}
Sep 16 17:57:31.262569 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.262527    2581 scope.go:117] "RemoveContainer" containerID="17f6aea38fcf71fd9aa1159b1d9a76a2be06b9db901832ccd95a29d301d12a4b"
Sep 16 17:57:31.273118 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.273096    2581 scope.go:117] "RemoveContainer" containerID="0dee1ed8439793f9397749643c258971fd3684f1722a51d1322f2fbdc727057d"
Sep 16 17:57:31.280649 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.280605    2581 scope.go:117] "RemoveContainer" containerID="c26c8edc73d33238a25f90dbaad2dfb5302cd8c61ad64dec560ef8d689e62231"
Sep 16 17:57:31.289055 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.289039    2581 scope.go:117] "RemoveContainer" containerID="17f6aea38fcf71fd9aa1159b1d9a76a2be06b9db901832ccd95a29d301d12a4b"
Sep 16 17:57:31.289342 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:57:31.289318    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17f6aea38fcf71fd9aa1159b1d9a76a2be06b9db901832ccd95a29d301d12a4b\": container with ID starting with 17f6aea38fcf71fd9aa1159b1d9a76a2be06b9db901832ccd95a29d301d12a4b not found: ID does not exist" containerID="17f6aea38fcf71fd9aa1159b1d9a76a2be06b9db901832ccd95a29d301d12a4b"
Sep 16 17:57:31.289413 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.289354    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17f6aea38fcf71fd9aa1159b1d9a76a2be06b9db901832ccd95a29d301d12a4b"} err="failed to get container status \"17f6aea38fcf71fd9aa1159b1d9a76a2be06b9db901832ccd95a29d301d12a4b\": rpc error: code = NotFound desc = could not find container \"17f6aea38fcf71fd9aa1159b1d9a76a2be06b9db901832ccd95a29d301d12a4b\": container with ID starting with 17f6aea38fcf71fd9aa1159b1d9a76a2be06b9db901832ccd95a29d301d12a4b not found: ID does not exist"
Sep 16 17:57:31.289413 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.289377    2581 scope.go:117] "RemoveContainer" containerID="0dee1ed8439793f9397749643c258971fd3684f1722a51d1322f2fbdc727057d"
Sep 16 17:57:31.289667 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:57:31.289647    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dee1ed8439793f9397749643c258971fd3684f1722a51d1322f2fbdc727057d\": container with ID starting with 0dee1ed8439793f9397749643c258971fd3684f1722a51d1322f2fbdc727057d not found: ID does not exist" containerID="0dee1ed8439793f9397749643c258971fd3684f1722a51d1322f2fbdc727057d"
Sep 16 17:57:31.289740 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.289672    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dee1ed8439793f9397749643c258971fd3684f1722a51d1322f2fbdc727057d"} err="failed to get container status \"0dee1ed8439793f9397749643c258971fd3684f1722a51d1322f2fbdc727057d\": rpc error: code = NotFound desc = could not find container \"0dee1ed8439793f9397749643c258971fd3684f1722a51d1322f2fbdc727057d\": container with ID starting with 0dee1ed8439793f9397749643c258971fd3684f1722a51d1322f2fbdc727057d not found: ID does not exist"
Sep 16 17:57:31.289740 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.289689    2581 scope.go:117] "RemoveContainer" containerID="c26c8edc73d33238a25f90dbaad2dfb5302cd8c61ad64dec560ef8d689e62231"
Sep 16 17:57:31.290001 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:57:31.289972    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c26c8edc73d33238a25f90dbaad2dfb5302cd8c61ad64dec560ef8d689e62231\": container with ID starting with c26c8edc73d33238a25f90dbaad2dfb5302cd8c61ad64dec560ef8d689e62231 not found: ID does not exist" containerID="c26c8edc73d33238a25f90dbaad2dfb5302cd8c61ad64dec560ef8d689e62231"
Sep 16 17:57:31.290084 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.290005    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c26c8edc73d33238a25f90dbaad2dfb5302cd8c61ad64dec560ef8d689e62231"} err="failed to get container status \"c26c8edc73d33238a25f90dbaad2dfb5302cd8c61ad64dec560ef8d689e62231\": rpc error: code = NotFound desc = could not find container \"c26c8edc73d33238a25f90dbaad2dfb5302cd8c61ad64dec560ef8d689e62231\": container with ID starting with c26c8edc73d33238a25f90dbaad2dfb5302cd8c61ad64dec560ef8d689e62231 not found: ID does not exist"
Sep 16 17:57:31.312372 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.312350    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kube-rbac-proxy-sar-config\") pod \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") "
Sep 16 17:57:31.312475 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.312397    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kserve-provision-location\") pod \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") "
Sep 16 17:57:31.312475 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.312433    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-proxy-tls\") pod \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") "
Sep 16 17:57:31.312475 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.312467    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvcd6\" (UniqueName: \"kubernetes.io/projected/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kube-api-access-bvcd6\") pod \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") "
Sep 16 17:57:31.312643 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.312502    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-cabundle-cert\") pod \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\" (UID: \"a8723ca3-5bc1-4199-88f3-edd9b413a3a7\") "
Sep 16 17:57:31.312740 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.312718    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a8723ca3-5bc1-4199-88f3-edd9b413a3a7" (UID: "a8723ca3-5bc1-4199-88f3-edd9b413a3a7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:57:31.312798 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.312770    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8723ca3-5bc1-4199-88f3-edd9b413a3a7" (UID: "a8723ca3-5bc1-4199-88f3-edd9b413a3a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:31.313077 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.313054    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a8723ca3-5bc1-4199-88f3-edd9b413a3a7" (UID: "a8723ca3-5bc1-4199-88f3-edd9b413a3a7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:57:31.314503 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.314477    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kube-api-access-bvcd6" (OuterVolumeSpecName: "kube-api-access-bvcd6") pod "a8723ca3-5bc1-4199-88f3-edd9b413a3a7" (UID: "a8723ca3-5bc1-4199-88f3-edd9b413a3a7"). InnerVolumeSpecName "kube-api-access-bvcd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:57:31.314607 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.314591    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8723ca3-5bc1-4199-88f3-edd9b413a3a7" (UID: "a8723ca3-5bc1-4199-88f3-edd9b413a3a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:57:31.414124 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.414085    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:57:31.414124 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.414121    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvcd6\" (UniqueName: \"kubernetes.io/projected/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kube-api-access-bvcd6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:57:31.414259 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.414135    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:57:31.414259 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.414147    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:57:31.414259 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.414160    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8723ca3-5bc1-4199-88f3-edd9b413a3a7-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:57:31.592214 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.592187    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"]
Sep 16 17:57:31.608881 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.608858    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-4rtq5"]
Sep 16 17:57:31.766042 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:31.766013    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" path="/var/lib/kubelet/pods/a8723ca3-5bc1-4199-88f3-edd9b413a3a7/volumes"
Sep 16 17:57:39.003759 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:39.003721    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 17:57:44.947285 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:44.947235    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:44.957203 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:44.957177    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"]
Sep 16 17:57:44.957420 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:44.957385    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kserve-container" containerID="cri-o://c17bc7550b4d8bea062c3909856a34d61c49d70497fa1024a4e5fae3f2479711" gracePeriod=30
Sep 16 17:57:44.957574 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:44.957419    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kube-rbac-proxy" containerID="cri-o://4a84943deecedc43c1d136497b23f345d60b52ae2678bb1429695b204766519b" gracePeriod=30
Sep 16 17:57:45.024031 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:45.024006    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:45.328766 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:45.328741    2581 generic.go:356] "Generic (PLEG): container finished" podID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerID="4a84943deecedc43c1d136497b23f345d60b52ae2678bb1429695b204766519b" exitCode=2
Sep 16 17:57:45.328873 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:45.328814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" event={"ID":"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a","Type":"ContainerDied","Data":"4a84943deecedc43c1d136497b23f345d60b52ae2678bb1429695b204766519b"}
Sep 16 17:57:46.887764 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:46.887728    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 16 17:57:46.892039 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:46.892006    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 17:57:48.697043 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.697022    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:57:48.834757 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.834731    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kserve-provision-location\") pod \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") "
Sep 16 17:57:48.834879 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.834767    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xglkb\" (UniqueName: \"kubernetes.io/projected/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kube-api-access-xglkb\") pod \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") "
Sep 16 17:57:48.834879 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.834817    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kube-rbac-proxy-sar-config\") pod \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") "
Sep 16 17:57:48.834879 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.834845    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-proxy-tls\") pod \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") "
Sep 16 17:57:48.834879 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.834873    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-cabundle-cert\") pod \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\" (UID: \"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a\") "
Sep 16 17:57:48.835088 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.835022    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" (UID: "cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:48.835196 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.835170    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" (UID: "cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:57:48.835335 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.835311    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" (UID: "cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:57:48.836956 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.836934    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" (UID: "cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:57:48.837072 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.837052    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kube-api-access-xglkb" (OuterVolumeSpecName: "kube-api-access-xglkb") pod "cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" (UID: "cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a"). InnerVolumeSpecName "kube-api-access-xglkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:57:48.935613 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.935586    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:57:48.935613 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.935608    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:57:48.935613 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.935617    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:57:48.935860 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.935625    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xglkb\" (UniqueName: \"kubernetes.io/projected/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kube-api-access-xglkb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:57:48.935860 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:48.935635    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:57:49.003940 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.003921    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:57:49.349729 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.349695    2581 generic.go:356] "Generic (PLEG): container finished" podID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerID="c17bc7550b4d8bea062c3909856a34d61c49d70497fa1024a4e5fae3f2479711" exitCode=0
Sep 16 17:57:49.349883 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.349756    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"
Sep 16 17:57:49.349883 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.349780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" event={"ID":"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a","Type":"ContainerDied","Data":"c17bc7550b4d8bea062c3909856a34d61c49d70497fa1024a4e5fae3f2479711"}
Sep 16 17:57:49.349883 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.349853    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs" event={"ID":"cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a","Type":"ContainerDied","Data":"bad93dc950535b251270adc3d9a3d383948db7e02a119c2174b39cd2b850d300"}
Sep 16 17:57:49.350095 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.349912    2581 scope.go:117] "RemoveContainer" containerID="4a84943deecedc43c1d136497b23f345d60b52ae2678bb1429695b204766519b"
Sep 16 17:57:49.359060 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.359043    2581 scope.go:117] "RemoveContainer" containerID="c17bc7550b4d8bea062c3909856a34d61c49d70497fa1024a4e5fae3f2479711"
Sep 16 17:57:49.366714 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.366695    2581 scope.go:117] "RemoveContainer" containerID="85e17cd370ef9b49d6a94e103ba0ff3a0efb5fc51a553e31b6ba9e4f4eb92b77"
Sep 16 17:57:49.374093 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.374075    2581 scope.go:117] "RemoveContainer" containerID="4a84943deecedc43c1d136497b23f345d60b52ae2678bb1429695b204766519b"
Sep 16 17:57:49.374347 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:57:49.374328    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a84943deecedc43c1d136497b23f345d60b52ae2678bb1429695b204766519b\": container with ID starting with 4a84943deecedc43c1d136497b23f345d60b52ae2678bb1429695b204766519b not found: ID does not exist" containerID="4a84943deecedc43c1d136497b23f345d60b52ae2678bb1429695b204766519b"
Sep 16 17:57:49.374389 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.374356    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a84943deecedc43c1d136497b23f345d60b52ae2678bb1429695b204766519b"} err="failed to get container status \"4a84943deecedc43c1d136497b23f345d60b52ae2678bb1429695b204766519b\": rpc error: code = NotFound desc = could not find container \"4a84943deecedc43c1d136497b23f345d60b52ae2678bb1429695b204766519b\": container with ID starting with 4a84943deecedc43c1d136497b23f345d60b52ae2678bb1429695b204766519b not found: ID does not exist"
Sep 16 17:57:49.374389 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.374372    2581 scope.go:117] "RemoveContainer" containerID="c17bc7550b4d8bea062c3909856a34d61c49d70497fa1024a4e5fae3f2479711"
Sep 16 17:57:49.374628 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:57:49.374611    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17bc7550b4d8bea062c3909856a34d61c49d70497fa1024a4e5fae3f2479711\": container with ID starting with c17bc7550b4d8bea062c3909856a34d61c49d70497fa1024a4e5fae3f2479711 not found: ID does not exist" containerID="c17bc7550b4d8bea062c3909856a34d61c49d70497fa1024a4e5fae3f2479711"
Sep 16 17:57:49.374679 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.374633    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17bc7550b4d8bea062c3909856a34d61c49d70497fa1024a4e5fae3f2479711"} err="failed to get container status \"c17bc7550b4d8bea062c3909856a34d61c49d70497fa1024a4e5fae3f2479711\": rpc error: code = NotFound desc = could not find container \"c17bc7550b4d8bea062c3909856a34d61c49d70497fa1024a4e5fae3f2479711\": container with ID starting with c17bc7550b4d8bea062c3909856a34d61c49d70497fa1024a4e5fae3f2479711 not found: ID does not exist"
Sep 16 17:57:49.374679 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.374648    2581 scope.go:117] "RemoveContainer" containerID="85e17cd370ef9b49d6a94e103ba0ff3a0efb5fc51a553e31b6ba9e4f4eb92b77"
Sep 16 17:57:49.374923 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:57:49.374883    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e17cd370ef9b49d6a94e103ba0ff3a0efb5fc51a553e31b6ba9e4f4eb92b77\": container with ID starting with 85e17cd370ef9b49d6a94e103ba0ff3a0efb5fc51a553e31b6ba9e4f4eb92b77 not found: ID does not exist" containerID="85e17cd370ef9b49d6a94e103ba0ff3a0efb5fc51a553e31b6ba9e4f4eb92b77"
Sep 16 17:57:49.374977 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.374927    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e17cd370ef9b49d6a94e103ba0ff3a0efb5fc51a553e31b6ba9e4f4eb92b77"} err="failed to get container status \"85e17cd370ef9b49d6a94e103ba0ff3a0efb5fc51a553e31b6ba9e4f4eb92b77\": rpc error: code = NotFound desc = could not find container \"85e17cd370ef9b49d6a94e103ba0ff3a0efb5fc51a553e31b6ba9e4f4eb92b77\": container with ID starting with 85e17cd370ef9b49d6a94e103ba0ff3a0efb5fc51a553e31b6ba9e4f4eb92b77 not found: ID does not exist"
Sep 16 17:57:49.404818 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.404799    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"]
Sep 16 17:57:49.422945 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.422922    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-s7kvs"]
Sep 16 17:57:49.766281 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:49.766226    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" path="/var/lib/kubelet/pods/cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a/volumes"
Sep 16 17:57:50.049251 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.049204    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:50.149845 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.149816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:50.229675 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.229651    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:50.239459 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239435    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"]
Sep 16 17:57:50.239765 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239752    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="storage-initializer"
Sep 16 17:57:50.239838 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239767    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="storage-initializer"
Sep 16 17:57:50.239838 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239777    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kserve-container"
Sep 16 17:57:50.239838 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239783    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kserve-container"
Sep 16 17:57:50.239838 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239792    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kube-rbac-proxy"
Sep 16 17:57:50.239838 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239797    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kube-rbac-proxy"
Sep 16 17:57:50.239838 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239804    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="storage-initializer"
Sep 16 17:57:50.239838 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239809    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="storage-initializer"
Sep 16 17:57:50.239838 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239817    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kserve-container"
Sep 16 17:57:50.239838 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239829    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kserve-container"
Sep 16 17:57:50.239838 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239837    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kube-rbac-proxy"
Sep 16 17:57:50.239838 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239841    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kube-rbac-proxy"
Sep 16 17:57:50.240406 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239901    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kube-rbac-proxy"
Sep 16 17:57:50.240406 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239908    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kserve-container"
Sep 16 17:57:50.240406 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239916    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb4c765c-d52d-4ab5-ae19-10afbc9f6a8a" containerName="kserve-container"
Sep 16 17:57:50.240406 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.239921    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8723ca3-5bc1-4199-88f3-edd9b413a3a7" containerName="kube-rbac-proxy"
Sep 16 17:57:50.244325 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.244309    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.247640 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.247621    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 17:57:50.247743 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.247651    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 17:57:50.265912 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.262536    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"]
Sep 16 17:57:50.345148 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.345097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swkhh\" (UniqueName: \"kubernetes.io/projected/bf846d93-f53a-4348-b622-5a6cdd1930e2-kube-api-access-swkhh\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-sj2rj\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.345148 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.345133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf846d93-f53a-4348-b622-5a6cdd1930e2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-sj2rj\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.345286 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.345165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf846d93-f53a-4348-b622-5a6cdd1930e2-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-sj2rj\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.345286 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.345204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf846d93-f53a-4348-b622-5a6cdd1930e2-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-sj2rj\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.345286 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.345244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf846d93-f53a-4348-b622-5a6cdd1930e2-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-sj2rj\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.429146 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.429118    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:50.446188 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.446167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swkhh\" (UniqueName: \"kubernetes.io/projected/bf846d93-f53a-4348-b622-5a6cdd1930e2-kube-api-access-swkhh\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-sj2rj\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.446277 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.446204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf846d93-f53a-4348-b622-5a6cdd1930e2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-sj2rj\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.446337 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.446316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf846d93-f53a-4348-b622-5a6cdd1930e2-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-sj2rj\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.446408 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.446372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf846d93-f53a-4348-b622-5a6cdd1930e2-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-sj2rj\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.446408 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.446401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf846d93-f53a-4348-b622-5a6cdd1930e2-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-sj2rj\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.446777 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.446760    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf846d93-f53a-4348-b622-5a6cdd1930e2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-sj2rj\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.446834 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.446764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf846d93-f53a-4348-b622-5a6cdd1930e2-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-sj2rj\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.447016 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.447002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf846d93-f53a-4348-b622-5a6cdd1930e2-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-sj2rj\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.448861 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.448844    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf846d93-f53a-4348-b622-5a6cdd1930e2-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-sj2rj\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.459652 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.459634    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swkhh\" (UniqueName: \"kubernetes.io/projected/bf846d93-f53a-4348-b622-5a6cdd1930e2-kube-api-access-swkhh\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-sj2rj\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.553804 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.553785    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:50.826729 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:50.826646    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"]
Sep 16 17:57:50.828930 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:57:50.828882    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf846d93_f53a_4348_b622_5a6cdd1930e2.slice/crio-ddec7b56ab9a7ae013651f058bb86f8cfdd3b0b2772cc8c0cb4bd684217bd1d9 WatchSource:0}: Error finding container ddec7b56ab9a7ae013651f058bb86f8cfdd3b0b2772cc8c0cb4bd684217bd1d9: Status 404 returned error can't find the container with id ddec7b56ab9a7ae013651f058bb86f8cfdd3b0b2772cc8c0cb4bd684217bd1d9
Sep 16 17:57:51.361649 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:51.361615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" event={"ID":"bf846d93-f53a-4348-b622-5a6cdd1930e2","Type":"ContainerStarted","Data":"f38c55e75d57d18b87b733b106be19371a00e7a6630f717488b6b6e5daa5dfba"}
Sep 16 17:57:51.361831 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:51.361670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" event={"ID":"bf846d93-f53a-4348-b622-5a6cdd1930e2","Type":"ContainerStarted","Data":"ddec7b56ab9a7ae013651f058bb86f8cfdd3b0b2772cc8c0cb4bd684217bd1d9"}
Sep 16 17:57:55.380150 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:55.380121    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerID="f38c55e75d57d18b87b733b106be19371a00e7a6630f717488b6b6e5daa5dfba" exitCode=0
Sep 16 17:57:55.380456 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:55.380197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" event={"ID":"bf846d93-f53a-4348-b622-5a6cdd1930e2","Type":"ContainerDied","Data":"f38c55e75d57d18b87b733b106be19371a00e7a6630f717488b6b6e5daa5dfba"}
Sep 16 17:57:56.387067 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:56.387032    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" event={"ID":"bf846d93-f53a-4348-b622-5a6cdd1930e2","Type":"ContainerStarted","Data":"2efb59ff7f8ec4102f8b8e26f59417d02543e8cc94753552a69ae9211b23b992"}
Sep 16 17:57:56.387428 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:56.387090    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" event={"ID":"bf846d93-f53a-4348-b622-5a6cdd1930e2","Type":"ContainerStarted","Data":"c7f24b116023b4f5af85f2cf2bc08567c5c35ec44b5fe073c366a7809515933b"}
Sep 16 17:57:56.387428 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:56.387358    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:56.412991 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:56.412952    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" podStartSLOduration=2.578214025 podStartE2EDuration="6.412939438s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.834725413s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:57:56.410646268 +0000 UTC m=+2005.199268802" watchObservedRunningTime="2026-09-16 17:57:56.412939438 +0000 UTC m=+2005.201561970"
Sep 16 17:57:56.988925 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:56.988879    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"]
Sep 16 17:57:56.989154 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:56.989117    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kserve-container" containerID="cri-o://949b5a7a132a0613da503988d4a0729ba4689b941fb0b95d2419b9e6df4dcfe9" gracePeriod=30
Sep 16 17:57:56.989225 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:56.989146    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kube-rbac-proxy" containerID="cri-o://121d2aacc6953a2d10137e6c0274256f681d2604b91389beb3188894a62df8ea" gracePeriod=30
Sep 16 17:57:57.029670 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:57.029636    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:57.393558 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:57.393531    2581 generic.go:356] "Generic (PLEG): container finished" podID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerID="121d2aacc6953a2d10137e6c0274256f681d2604b91389beb3188894a62df8ea" exitCode=2
Sep 16 17:57:57.393968 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:57.393599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" event={"ID":"5248c29d-bb8a-4a81-8984-4fbecd036a2d","Type":"ContainerDied","Data":"121d2aacc6953a2d10137e6c0274256f681d2604b91389beb3188894a62df8ea"}
Sep 16 17:57:57.393968 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:57.393884    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:57:57.395073 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:57.395056    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 17:57:58.399242 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:58.399204    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 17:57:58.999368 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:58.999330    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 16 17:57:59.003592 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:57:59.003571    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 17:58:00.825092 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:00.825068    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:58:00.929934 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:00.929868    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kube-rbac-proxy-sar-config\") pod \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") "
Sep 16 17:58:00.930037 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:00.929971    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kserve-provision-location\") pod \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") "
Sep 16 17:58:00.930037 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:00.929995    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5248c29d-bb8a-4a81-8984-4fbecd036a2d-cabundle-cert\") pod \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") "
Sep 16 17:58:00.930037 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:00.930017    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5248c29d-bb8a-4a81-8984-4fbecd036a2d-proxy-tls\") pod \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") "
Sep 16 17:58:00.930153 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:00.930051    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znsvs\" (UniqueName: \"kubernetes.io/projected/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kube-api-access-znsvs\") pod \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\" (UID: \"5248c29d-bb8a-4a81-8984-4fbecd036a2d\") "
Sep 16 17:58:00.930297 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:00.930263    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5248c29d-bb8a-4a81-8984-4fbecd036a2d" (UID: "5248c29d-bb8a-4a81-8984-4fbecd036a2d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:58:00.930405 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:00.930304    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5248c29d-bb8a-4a81-8984-4fbecd036a2d" (UID: "5248c29d-bb8a-4a81-8984-4fbecd036a2d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:58:00.930405 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:00.930319    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5248c29d-bb8a-4a81-8984-4fbecd036a2d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5248c29d-bb8a-4a81-8984-4fbecd036a2d" (UID: "5248c29d-bb8a-4a81-8984-4fbecd036a2d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:58:00.932264 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:00.932239    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kube-api-access-znsvs" (OuterVolumeSpecName: "kube-api-access-znsvs") pod "5248c29d-bb8a-4a81-8984-4fbecd036a2d" (UID: "5248c29d-bb8a-4a81-8984-4fbecd036a2d"). InnerVolumeSpecName "kube-api-access-znsvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:58:00.932352 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:00.932305    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5248c29d-bb8a-4a81-8984-4fbecd036a2d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5248c29d-bb8a-4a81-8984-4fbecd036a2d" (UID: "5248c29d-bb8a-4a81-8984-4fbecd036a2d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:58:01.031145 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.031120    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:58:01.031145 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.031142    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5248c29d-bb8a-4a81-8984-4fbecd036a2d-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:58:01.031261 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.031152    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5248c29d-bb8a-4a81-8984-4fbecd036a2d-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:58:01.031261 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.031159    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znsvs\" (UniqueName: \"kubernetes.io/projected/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kube-api-access-znsvs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:58:01.031261 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.031168    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5248c29d-bb8a-4a81-8984-4fbecd036a2d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:58:01.419094 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.419061    2581 generic.go:356] "Generic (PLEG): container finished" podID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerID="949b5a7a132a0613da503988d4a0729ba4689b941fb0b95d2419b9e6df4dcfe9" exitCode=0
Sep 16 17:58:01.419220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.419121    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"
Sep 16 17:58:01.419220 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.419149    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" event={"ID":"5248c29d-bb8a-4a81-8984-4fbecd036a2d","Type":"ContainerDied","Data":"949b5a7a132a0613da503988d4a0729ba4689b941fb0b95d2419b9e6df4dcfe9"}
Sep 16 17:58:01.419292 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.419231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6" event={"ID":"5248c29d-bb8a-4a81-8984-4fbecd036a2d","Type":"ContainerDied","Data":"0b6c1a8d53f01ba43669b3f9e1c0a5e9e6b018669f91755ee4d86f404045f24d"}
Sep 16 17:58:01.419292 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.419260    2581 scope.go:117] "RemoveContainer" containerID="121d2aacc6953a2d10137e6c0274256f681d2604b91389beb3188894a62df8ea"
Sep 16 17:58:01.427782 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.427767    2581 scope.go:117] "RemoveContainer" containerID="949b5a7a132a0613da503988d4a0729ba4689b941fb0b95d2419b9e6df4dcfe9"
Sep 16 17:58:01.434919 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.434903    2581 scope.go:117] "RemoveContainer" containerID="05b994996bb6cb552149db5902a223c9d789b4171e8d6762cc73a37ec809751b"
Sep 16 17:58:01.442603 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.442581    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"]
Sep 16 17:58:01.443087 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.443071    2581 scope.go:117] "RemoveContainer" containerID="121d2aacc6953a2d10137e6c0274256f681d2604b91389beb3188894a62df8ea"
Sep 16 17:58:01.443362 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:58:01.443342    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121d2aacc6953a2d10137e6c0274256f681d2604b91389beb3188894a62df8ea\": container with ID starting with 121d2aacc6953a2d10137e6c0274256f681d2604b91389beb3188894a62df8ea not found: ID does not exist" containerID="121d2aacc6953a2d10137e6c0274256f681d2604b91389beb3188894a62df8ea"
Sep 16 17:58:01.443406 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.443373    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121d2aacc6953a2d10137e6c0274256f681d2604b91389beb3188894a62df8ea"} err="failed to get container status \"121d2aacc6953a2d10137e6c0274256f681d2604b91389beb3188894a62df8ea\": rpc error: code = NotFound desc = could not find container \"121d2aacc6953a2d10137e6c0274256f681d2604b91389beb3188894a62df8ea\": container with ID starting with 121d2aacc6953a2d10137e6c0274256f681d2604b91389beb3188894a62df8ea not found: ID does not exist"
Sep 16 17:58:01.443406 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.443389    2581 scope.go:117] "RemoveContainer" containerID="949b5a7a132a0613da503988d4a0729ba4689b941fb0b95d2419b9e6df4dcfe9"
Sep 16 17:58:01.443619 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:58:01.443601    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949b5a7a132a0613da503988d4a0729ba4689b941fb0b95d2419b9e6df4dcfe9\": container with ID starting with 949b5a7a132a0613da503988d4a0729ba4689b941fb0b95d2419b9e6df4dcfe9 not found: ID does not exist" containerID="949b5a7a132a0613da503988d4a0729ba4689b941fb0b95d2419b9e6df4dcfe9"
Sep 16 17:58:01.443665 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.443625    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949b5a7a132a0613da503988d4a0729ba4689b941fb0b95d2419b9e6df4dcfe9"} err="failed to get container status \"949b5a7a132a0613da503988d4a0729ba4689b941fb0b95d2419b9e6df4dcfe9\": rpc error: code = NotFound desc = could not find container \"949b5a7a132a0613da503988d4a0729ba4689b941fb0b95d2419b9e6df4dcfe9\": container with ID starting with 949b5a7a132a0613da503988d4a0729ba4689b941fb0b95d2419b9e6df4dcfe9 not found: ID does not exist"
Sep 16 17:58:01.443665 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.443639    2581 scope.go:117] "RemoveContainer" containerID="05b994996bb6cb552149db5902a223c9d789b4171e8d6762cc73a37ec809751b"
Sep 16 17:58:01.443859 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:58:01.443843    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b994996bb6cb552149db5902a223c9d789b4171e8d6762cc73a37ec809751b\": container with ID starting with 05b994996bb6cb552149db5902a223c9d789b4171e8d6762cc73a37ec809751b not found: ID does not exist" containerID="05b994996bb6cb552149db5902a223c9d789b4171e8d6762cc73a37ec809751b"
Sep 16 17:58:01.443934 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.443866    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b994996bb6cb552149db5902a223c9d789b4171e8d6762cc73a37ec809751b"} err="failed to get container status \"05b994996bb6cb552149db5902a223c9d789b4171e8d6762cc73a37ec809751b\": rpc error: code = NotFound desc = could not find container \"05b994996bb6cb552149db5902a223c9d789b4171e8d6762cc73a37ec809751b\": container with ID starting with 05b994996bb6cb552149db5902a223c9d789b4171e8d6762cc73a37ec809751b not found: ID does not exist"
Sep 16 17:58:01.449241 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.449222    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-77fd7b9f9f-tljf6"]
Sep 16 17:58:01.765593 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:01.765541    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" path="/var/lib/kubelet/pods/5248c29d-bb8a-4a81-8984-4fbecd036a2d/volumes"
Sep 16 17:58:02.023663 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.023635    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:02.065372 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.065349    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"]
Sep 16 17:58:02.065703 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.065688    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kserve-container"
Sep 16 17:58:02.065754 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.065705    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kserve-container"
Sep 16 17:58:02.065754 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.065714    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kube-rbac-proxy"
Sep 16 17:58:02.065754 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.065720    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kube-rbac-proxy"
Sep 16 17:58:02.065754 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.065727    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="storage-initializer"
Sep 16 17:58:02.065754 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.065732    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="storage-initializer"
Sep 16 17:58:02.065941 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.065803    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kube-rbac-proxy"
Sep 16 17:58:02.065941 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.065816    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5248c29d-bb8a-4a81-8984-4fbecd036a2d" containerName="kserve-container"
Sep 16 17:58:02.070543 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.070527    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.073079 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.073051    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 16 17:58:02.073937 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.073921    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 16 17:58:02.082745 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.082725    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"]
Sep 16 17:58:02.212096 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.212073    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:02.240435 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.240411    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f759eb5-be9f-4666-8107-f47d53c21045-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-cr4fr\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.240533 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.240453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4km85\" (UniqueName: \"kubernetes.io/projected/4f759eb5-be9f-4666-8107-f47d53c21045-kube-api-access-4km85\") pod \"isvc-tensorflow-predictor-547dfc5669-cr4fr\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.240533 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.240478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f759eb5-be9f-4666-8107-f47d53c21045-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-cr4fr\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.240533 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.240513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f759eb5-be9f-4666-8107-f47d53c21045-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-cr4fr\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.240650 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.240536    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f759eb5-be9f-4666-8107-f47d53c21045-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-cr4fr\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.341711 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.341657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f759eb5-be9f-4666-8107-f47d53c21045-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-cr4fr\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.341711 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.341694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4km85\" (UniqueName: \"kubernetes.io/projected/4f759eb5-be9f-4666-8107-f47d53c21045-kube-api-access-4km85\") pod \"isvc-tensorflow-predictor-547dfc5669-cr4fr\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.341711 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.341710    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f759eb5-be9f-4666-8107-f47d53c21045-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-cr4fr\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.341918 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.341733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f759eb5-be9f-4666-8107-f47d53c21045-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-cr4fr\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.341918 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.341770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f759eb5-be9f-4666-8107-f47d53c21045-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-cr4fr\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.342123 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.342104    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f759eb5-be9f-4666-8107-f47d53c21045-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-cr4fr\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.342320 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.342304    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f759eb5-be9f-4666-8107-f47d53c21045-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-cr4fr\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.342376 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.342333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f759eb5-be9f-4666-8107-f47d53c21045-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-cr4fr\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.344395 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.344371    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f759eb5-be9f-4666-8107-f47d53c21045-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-cr4fr\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.350668 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.350644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4km85\" (UniqueName: \"kubernetes.io/projected/4f759eb5-be9f-4666-8107-f47d53c21045-kube-api-access-4km85\") pod \"isvc-tensorflow-predictor-547dfc5669-cr4fr\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.380401 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.380379    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:02.651179 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:02.651005    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"]
Sep 16 17:58:02.653769 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:58:02.653744    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f759eb5_be9f_4666_8107_f47d53c21045.slice/crio-fe6209215cee4b4273ecc66b927512c2b7ac1cf0006a9d2ef944a4c93741e913 WatchSource:0}: Error finding container fe6209215cee4b4273ecc66b927512c2b7ac1cf0006a9d2ef944a4c93741e913: Status 404 returned error can't find the container with id fe6209215cee4b4273ecc66b927512c2b7ac1cf0006a9d2ef944a4c93741e913
Sep 16 17:58:03.403884 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:03.403853    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:58:03.404469 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:03.404439    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 17:58:03.430712 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:03.430684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" event={"ID":"4f759eb5-be9f-4666-8107-f47d53c21045","Type":"ContainerStarted","Data":"76a6b5bb8acd2dcdd4775129ccbb1a2d573b21f908874290aa59af3ccebad59f"}
Sep 16 17:58:03.430822 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:03.430730    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" event={"ID":"4f759eb5-be9f-4666-8107-f47d53c21045","Type":"ContainerStarted","Data":"fe6209215cee4b4273ecc66b927512c2b7ac1cf0006a9d2ef944a4c93741e913"}
Sep 16 17:58:08.453482 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:08.453455    2581 generic.go:356] "Generic (PLEG): container finished" podID="4f759eb5-be9f-4666-8107-f47d53c21045" containerID="76a6b5bb8acd2dcdd4775129ccbb1a2d573b21f908874290aa59af3ccebad59f" exitCode=0
Sep 16 17:58:08.453745 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:08.453508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" event={"ID":"4f759eb5-be9f-4666-8107-f47d53c21045","Type":"ContainerDied","Data":"76a6b5bb8acd2dcdd4775129ccbb1a2d573b21f908874290aa59af3ccebad59f"}
Sep 16 17:58:08.454745 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:08.454728    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:58:08.548226 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:08.548201    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:12.476526 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:12.476488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" event={"ID":"4f759eb5-be9f-4666-8107-f47d53c21045","Type":"ContainerStarted","Data":"195777db0986149931a830bb437b2e3cdf1c530fe047bc64bdd710b1dd12cae2"}
Sep 16 17:58:12.476951 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:12.476574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" event={"ID":"4f759eb5-be9f-4666-8107-f47d53c21045","Type":"ContainerStarted","Data":"251c12b8ffdb22a1a9948f8acb3c06cacc8d94e5043803864ead0890f97da737"}
Sep 16 17:58:12.476951 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:12.476844    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:12.476951 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:12.476914    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:12.478216 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:12.478190    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 17:58:12.508791 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:12.508748    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" podStartSLOduration=1.327896801 podStartE2EDuration="10.508736607s" totalImagesPullingTime="3.535167679s" totalInitContainerRuntime="5.645672127s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:58:12.506740775 +0000 UTC m=+2021.295363307" watchObservedRunningTime="2026-09-16 17:58:12.508736607 +0000 UTC m=+2021.297359138"
Sep 16 17:58:13.404782 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:13.404747    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 17:58:13.481265 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:13.481229    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 17:58:18.485694 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:18.485668    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:18.486175 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:18.486152    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 17:58:23.404446 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:23.404409    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 17:58:28.486990 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:28.486961    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:33.404414 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:33.404377    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 17:58:40.886977 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:40.886936    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"]
Sep 16 17:58:40.887431 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:40.887234    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kserve-container" containerID="cri-o://251c12b8ffdb22a1a9948f8acb3c06cacc8d94e5043803864ead0890f97da737" gracePeriod=30
Sep 16 17:58:40.887431 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:40.887328    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kube-rbac-proxy" containerID="cri-o://195777db0986149931a830bb437b2e3cdf1c530fe047bc64bdd710b1dd12cae2" gracePeriod=30
Sep 16 17:58:40.941686 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:40.941663    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:41.605858 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:41.605826    2581 generic.go:356] "Generic (PLEG): container finished" podID="4f759eb5-be9f-4666-8107-f47d53c21045" containerID="195777db0986149931a830bb437b2e3cdf1c530fe047bc64bdd710b1dd12cae2" exitCode=2
Sep 16 17:58:41.606045 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:41.605918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" event={"ID":"4f759eb5-be9f-4666-8107-f47d53c21045","Type":"ContainerDied","Data":"195777db0986149931a830bb437b2e3cdf1c530fe047bc64bdd710b1dd12cae2"}
Sep 16 17:58:43.404492 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:43.404455    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 17:58:43.481911 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:43.481860    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 16 17:58:46.010698 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.010664    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:46.107792 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.107766    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"]
Sep 16 17:58:46.111831 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.111813    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.116816 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.116797    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 16 17:58:46.117053 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.117033    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 16 17:58:46.132988 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.132964    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"]
Sep 16 17:58:46.146405 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.146384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-8q2md\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.146504 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.146424    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjkjk\" (UniqueName: \"kubernetes.io/projected/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kube-api-access-hjkjk\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-8q2md\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.146504 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.146491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-8q2md\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.146594 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.146508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-8q2md\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.146633 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.146599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-8q2md\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.247527 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.247502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-8q2md\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.247660 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.247547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-8q2md\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.247660 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.247586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjkjk\" (UniqueName: \"kubernetes.io/projected/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kube-api-access-hjkjk\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-8q2md\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.247660 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.247649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-8q2md\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.247839 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.247673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-8q2md\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.248071 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.248043    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-8q2md\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.248308 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.248287    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-8q2md\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.248525 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.248503    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-8q2md\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.250654 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.250638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-8q2md\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.258252 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.258232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjkjk\" (UniqueName: \"kubernetes.io/projected/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kube-api-access-hjkjk\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-8q2md\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.275824 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.275801    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:46.423088 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.423059    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:46.727777 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:58:46.727735    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0aa9912_225e_47f9_a3f1_b89d3a4edf17.slice/crio-1f873a1ded6dbf9b10bf6c57f3f972a04cd6b3cd63f8cb6e8ffc62ea2ffcfb6b WatchSource:0}: Error finding container 1f873a1ded6dbf9b10bf6c57f3f972a04cd6b3cd63f8cb6e8ffc62ea2ffcfb6b: Status 404 returned error can't find the container with id 1f873a1ded6dbf9b10bf6c57f3f972a04cd6b3cd63f8cb6e8ffc62ea2ffcfb6b
Sep 16 17:58:46.728459 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:46.728434    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"]
Sep 16 17:58:47.650518 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:47.650482    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" event={"ID":"a0aa9912-225e-47f9-a3f1-b89d3a4edf17","Type":"ContainerStarted","Data":"f598e834d87c38bf8476fcb3b050d2c231e790b25a668718f66d1f31d9abe371"}
Sep 16 17:58:47.650880 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:47.650574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" event={"ID":"a0aa9912-225e-47f9-a3f1-b89d3a4edf17","Type":"ContainerStarted","Data":"1f873a1ded6dbf9b10bf6c57f3f972a04cd6b3cd63f8cb6e8ffc62ea2ffcfb6b"}
Sep 16 17:58:48.481964 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:48.481933    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 16 17:58:51.672208 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:51.672181    2581 generic.go:356] "Generic (PLEG): container finished" podID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerID="f598e834d87c38bf8476fcb3b050d2c231e790b25a668718f66d1f31d9abe371" exitCode=0
Sep 16 17:58:51.672574 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:51.672231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" event={"ID":"a0aa9912-225e-47f9-a3f1-b89d3a4edf17","Type":"ContainerDied","Data":"f598e834d87c38bf8476fcb3b050d2c231e790b25a668718f66d1f31d9abe371"}
Sep 16 17:58:52.679302 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:52.679220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" event={"ID":"a0aa9912-225e-47f9-a3f1-b89d3a4edf17","Type":"ContainerStarted","Data":"2860d61b913dc1e33e3728528fe359be37b7f9a51687d2c759c5d5cb52408863"}
Sep 16 17:58:52.679661 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:52.679302    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" event={"ID":"a0aa9912-225e-47f9-a3f1-b89d3a4edf17","Type":"ContainerStarted","Data":"8659c528736464a2068eb98a43ba71edf564edd14e4079df6662fd14f03a667c"}
Sep 16 17:58:52.679661 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:52.679505    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:52.714591 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:52.714537    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" podStartSLOduration=2.343361748 podStartE2EDuration="6.714519067s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.371157319s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:58:52.713110324 +0000 UTC m=+2061.501732857" watchObservedRunningTime="2026-09-16 17:58:52.714519067 +0000 UTC m=+2061.503141600"
Sep 16 17:58:53.405142 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:53.405103    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 17:58:53.482258 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:53.482219    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 16 17:58:53.482419 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:53.482342    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:58:53.684230 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:53.684149    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:53.685340 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:53.685311    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 17:58:54.688695 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:54.688656    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 17:58:58.481852 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:58.481809    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 16 17:58:59.698100 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:59.698069    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:58:59.698609 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:58:59.698581    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 17:59:03.404786 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:03.404759    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:59:03.481993 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:03.481955    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 16 17:59:08.482449 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:08.482411    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 16 17:59:09.699585 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:09.699558    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:59:11.527798 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.527768    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:59:11.627269 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.627244    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4km85\" (UniqueName: \"kubernetes.io/projected/4f759eb5-be9f-4666-8107-f47d53c21045-kube-api-access-4km85\") pod \"4f759eb5-be9f-4666-8107-f47d53c21045\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") "
Sep 16 17:59:11.627375 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.627279    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f759eb5-be9f-4666-8107-f47d53c21045-kserve-provision-location\") pod \"4f759eb5-be9f-4666-8107-f47d53c21045\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") "
Sep 16 17:59:11.627375 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.627297    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f759eb5-be9f-4666-8107-f47d53c21045-proxy-tls\") pod \"4f759eb5-be9f-4666-8107-f47d53c21045\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") "
Sep 16 17:59:11.627484 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.627376    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f759eb5-be9f-4666-8107-f47d53c21045-kube-rbac-proxy-sar-config\") pod \"4f759eb5-be9f-4666-8107-f47d53c21045\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") "
Sep 16 17:59:11.627484 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.627394    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f759eb5-be9f-4666-8107-f47d53c21045-cabundle-cert\") pod \"4f759eb5-be9f-4666-8107-f47d53c21045\" (UID: \"4f759eb5-be9f-4666-8107-f47d53c21045\") "
Sep 16 17:59:11.627734 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.627705    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f759eb5-be9f-4666-8107-f47d53c21045-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f759eb5-be9f-4666-8107-f47d53c21045" (UID: "4f759eb5-be9f-4666-8107-f47d53c21045"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:59:11.627824 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.627733    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f759eb5-be9f-4666-8107-f47d53c21045-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f759eb5-be9f-4666-8107-f47d53c21045" (UID: "4f759eb5-be9f-4666-8107-f47d53c21045"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:59:11.629406 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.629384    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f759eb5-be9f-4666-8107-f47d53c21045-kube-api-access-4km85" (OuterVolumeSpecName: "kube-api-access-4km85") pod "4f759eb5-be9f-4666-8107-f47d53c21045" (UID: "4f759eb5-be9f-4666-8107-f47d53c21045"). InnerVolumeSpecName "kube-api-access-4km85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:59:11.629545 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.629523    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f759eb5-be9f-4666-8107-f47d53c21045-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f759eb5-be9f-4666-8107-f47d53c21045" (UID: "4f759eb5-be9f-4666-8107-f47d53c21045"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:59:11.637861 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.637838    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f759eb5-be9f-4666-8107-f47d53c21045-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f759eb5-be9f-4666-8107-f47d53c21045" (UID: "4f759eb5-be9f-4666-8107-f47d53c21045"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:11.728463 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.728416    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f759eb5-be9f-4666-8107-f47d53c21045-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:59:11.728463 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.728435    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f759eb5-be9f-4666-8107-f47d53c21045-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:59:11.728463 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.728444    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4km85\" (UniqueName: \"kubernetes.io/projected/4f759eb5-be9f-4666-8107-f47d53c21045-kube-api-access-4km85\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:59:11.728463 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.728453    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f759eb5-be9f-4666-8107-f47d53c21045-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:59:11.728463 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.728462    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f759eb5-be9f-4666-8107-f47d53c21045-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:59:11.768929 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.768883    2581 generic.go:356] "Generic (PLEG): container finished" podID="4f759eb5-be9f-4666-8107-f47d53c21045" containerID="251c12b8ffdb22a1a9948f8acb3c06cacc8d94e5043803864ead0890f97da737" exitCode=137
Sep 16 17:59:11.769012 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.768961    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"
Sep 16 17:59:11.769012 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.768990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" event={"ID":"4f759eb5-be9f-4666-8107-f47d53c21045","Type":"ContainerDied","Data":"251c12b8ffdb22a1a9948f8acb3c06cacc8d94e5043803864ead0890f97da737"}
Sep 16 17:59:11.769082 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.769055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr" event={"ID":"4f759eb5-be9f-4666-8107-f47d53c21045","Type":"ContainerDied","Data":"fe6209215cee4b4273ecc66b927512c2b7ac1cf0006a9d2ef944a4c93741e913"}
Sep 16 17:59:11.769115 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.769097    2581 scope.go:117] "RemoveContainer" containerID="195777db0986149931a830bb437b2e3cdf1c530fe047bc64bdd710b1dd12cae2"
Sep 16 17:59:11.778150 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.778130    2581 scope.go:117] "RemoveContainer" containerID="251c12b8ffdb22a1a9948f8acb3c06cacc8d94e5043803864ead0890f97da737"
Sep 16 17:59:11.785403 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.785384    2581 scope.go:117] "RemoveContainer" containerID="76a6b5bb8acd2dcdd4775129ccbb1a2d573b21f908874290aa59af3ccebad59f"
Sep 16 17:59:11.792295 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.792279    2581 scope.go:117] "RemoveContainer" containerID="195777db0986149931a830bb437b2e3cdf1c530fe047bc64bdd710b1dd12cae2"
Sep 16 17:59:11.792573 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:59:11.792554    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195777db0986149931a830bb437b2e3cdf1c530fe047bc64bdd710b1dd12cae2\": container with ID starting with 195777db0986149931a830bb437b2e3cdf1c530fe047bc64bdd710b1dd12cae2 not found: ID does not exist" containerID="195777db0986149931a830bb437b2e3cdf1c530fe047bc64bdd710b1dd12cae2"
Sep 16 17:59:11.792663 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.792578    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195777db0986149931a830bb437b2e3cdf1c530fe047bc64bdd710b1dd12cae2"} err="failed to get container status \"195777db0986149931a830bb437b2e3cdf1c530fe047bc64bdd710b1dd12cae2\": rpc error: code = NotFound desc = could not find container \"195777db0986149931a830bb437b2e3cdf1c530fe047bc64bdd710b1dd12cae2\": container with ID starting with 195777db0986149931a830bb437b2e3cdf1c530fe047bc64bdd710b1dd12cae2 not found: ID does not exist"
Sep 16 17:59:11.792663 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.792593    2581 scope.go:117] "RemoveContainer" containerID="251c12b8ffdb22a1a9948f8acb3c06cacc8d94e5043803864ead0890f97da737"
Sep 16 17:59:11.792852 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:59:11.792818    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"251c12b8ffdb22a1a9948f8acb3c06cacc8d94e5043803864ead0890f97da737\": container with ID starting with 251c12b8ffdb22a1a9948f8acb3c06cacc8d94e5043803864ead0890f97da737 not found: ID does not exist" containerID="251c12b8ffdb22a1a9948f8acb3c06cacc8d94e5043803864ead0890f97da737"
Sep 16 17:59:11.792852 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.792845    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251c12b8ffdb22a1a9948f8acb3c06cacc8d94e5043803864ead0890f97da737"} err="failed to get container status \"251c12b8ffdb22a1a9948f8acb3c06cacc8d94e5043803864ead0890f97da737\": rpc error: code = NotFound desc = could not find container \"251c12b8ffdb22a1a9948f8acb3c06cacc8d94e5043803864ead0890f97da737\": container with ID starting with 251c12b8ffdb22a1a9948f8acb3c06cacc8d94e5043803864ead0890f97da737 not found: ID does not exist"
Sep 16 17:59:11.792957 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.792857    2581 scope.go:117] "RemoveContainer" containerID="76a6b5bb8acd2dcdd4775129ccbb1a2d573b21f908874290aa59af3ccebad59f"
Sep 16 17:59:11.793166 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:59:11.793141    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a6b5bb8acd2dcdd4775129ccbb1a2d573b21f908874290aa59af3ccebad59f\": container with ID starting with 76a6b5bb8acd2dcdd4775129ccbb1a2d573b21f908874290aa59af3ccebad59f not found: ID does not exist" containerID="76a6b5bb8acd2dcdd4775129ccbb1a2d573b21f908874290aa59af3ccebad59f"
Sep 16 17:59:11.793217 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.793184    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a6b5bb8acd2dcdd4775129ccbb1a2d573b21f908874290aa59af3ccebad59f"} err="failed to get container status \"76a6b5bb8acd2dcdd4775129ccbb1a2d573b21f908874290aa59af3ccebad59f\": rpc error: code = NotFound desc = could not find container \"76a6b5bb8acd2dcdd4775129ccbb1a2d573b21f908874290aa59af3ccebad59f\": container with ID starting with 76a6b5bb8acd2dcdd4775129ccbb1a2d573b21f908874290aa59af3ccebad59f not found: ID does not exist"
Sep 16 17:59:11.796148 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.796128    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"]
Sep 16 17:59:11.804693 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:11.804673    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cr4fr"]
Sep 16 17:59:13.766140 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:13.766109    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" path="/var/lib/kubelet/pods/4f759eb5-be9f-4666-8107-f47d53c21045/volumes"
Sep 16 17:59:20.161473 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.161437    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"]
Sep 16 17:59:20.161853 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.161664    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kserve-container" containerID="cri-o://c7f24b116023b4f5af85f2cf2bc08567c5c35ec44b5fe073c366a7809515933b" gracePeriod=30
Sep 16 17:59:20.161853 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.161710    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kube-rbac-proxy" containerID="cri-o://2efb59ff7f8ec4102f8b8e26f59417d02543e8cc94753552a69ae9211b23b992" gracePeriod=30
Sep 16 17:59:20.222187 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.222161    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:20.367161 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.367134    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"]
Sep 16 17:59:20.367456 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.367444    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kserve-container"
Sep 16 17:59:20.367456 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.367457    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kserve-container"
Sep 16 17:59:20.367567 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.367474    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="storage-initializer"
Sep 16 17:59:20.367567 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.367479    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="storage-initializer"
Sep 16 17:59:20.367567 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.367495    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kube-rbac-proxy"
Sep 16 17:59:20.367567 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.367500    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kube-rbac-proxy"
Sep 16 17:59:20.367567 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.367550    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kube-rbac-proxy"
Sep 16 17:59:20.367567 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.367560    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f759eb5-be9f-4666-8107-f47d53c21045" containerName="kserve-container"
Sep 16 17:59:20.370656 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.370639    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.374214 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.374187    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 17:59:20.393910 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.393865    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"]
Sep 16 17:59:20.488811 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.488749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnnrm\" (UniqueName: \"kubernetes.io/projected/03fb78d7-5343-4534-8f6f-273a6221a8d1-kube-api-access-rnnrm\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-fr64s\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.488811 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.488787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03fb78d7-5343-4534-8f6f-273a6221a8d1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-fr64s\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.488970 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.488876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03fb78d7-5343-4534-8f6f-273a6221a8d1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-fr64s\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.488970 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.488958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03fb78d7-5343-4534-8f6f-273a6221a8d1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-fr64s\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.489036 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.489000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03fb78d7-5343-4534-8f6f-273a6221a8d1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-fr64s\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.589787 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.589759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03fb78d7-5343-4534-8f6f-273a6221a8d1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-fr64s\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.589927 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.589810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03fb78d7-5343-4534-8f6f-273a6221a8d1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-fr64s\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.589927 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.589866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03fb78d7-5343-4534-8f6f-273a6221a8d1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-fr64s\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.590054 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.589940    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnnrm\" (UniqueName: \"kubernetes.io/projected/03fb78d7-5343-4534-8f6f-273a6221a8d1-kube-api-access-rnnrm\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-fr64s\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.590054 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.589969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03fb78d7-5343-4534-8f6f-273a6221a8d1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-fr64s\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.590169 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.590148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03fb78d7-5343-4534-8f6f-273a6221a8d1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-fr64s\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.590515 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.590494    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03fb78d7-5343-4534-8f6f-273a6221a8d1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-fr64s\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.590584 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.590515    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03fb78d7-5343-4534-8f6f-273a6221a8d1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-fr64s\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.592424 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.592405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03fb78d7-5343-4534-8f6f-273a6221a8d1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-fr64s\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.603859 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.603838    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnnrm\" (UniqueName: \"kubernetes.io/projected/03fb78d7-5343-4534-8f6f-273a6221a8d1-kube-api-access-rnnrm\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-fr64s\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.679829 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.679804    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:20.809879 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.809851    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerID="2efb59ff7f8ec4102f8b8e26f59417d02543e8cc94753552a69ae9211b23b992" exitCode=2
Sep 16 17:59:20.810041 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.809935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" event={"ID":"bf846d93-f53a-4348-b622-5a6cdd1930e2","Type":"ContainerDied","Data":"2efb59ff7f8ec4102f8b8e26f59417d02543e8cc94753552a69ae9211b23b992"}
Sep 16 17:59:20.956028 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:20.955985    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"]
Sep 16 17:59:20.958181 ip-10-0-128-31 kubenswrapper[2581]: W0916 17:59:20.958154    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03fb78d7_5343_4534_8f6f_273a6221a8d1.slice/crio-106f4547fae45f2d458a488efbe0a0e909057ea5075405044ed0a38a72509c59 WatchSource:0}: Error finding container 106f4547fae45f2d458a488efbe0a0e909057ea5075405044ed0a38a72509c59: Status 404 returned error can't find the container with id 106f4547fae45f2d458a488efbe0a0e909057ea5075405044ed0a38a72509c59
Sep 16 17:59:21.815580 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:21.815544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" event={"ID":"03fb78d7-5343-4534-8f6f-273a6221a8d1","Type":"ContainerStarted","Data":"3491bfb61bfdaf9f46683c1c8044f55f890f6194c50638c58fe02329188dfb11"}
Sep 16 17:59:21.816065 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:21.815604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" event={"ID":"03fb78d7-5343-4534-8f6f-273a6221a8d1","Type":"ContainerStarted","Data":"106f4547fae45f2d458a488efbe0a0e909057ea5075405044ed0a38a72509c59"}
Sep 16 17:59:23.399825 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:23.399774    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 16 17:59:23.405087 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:23.405063    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 17:59:24.301737 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.301716    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:59:24.420598 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.420530    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf846d93-f53a-4348-b622-5a6cdd1930e2-kube-rbac-proxy-sar-config\") pod \"bf846d93-f53a-4348-b622-5a6cdd1930e2\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") "
Sep 16 17:59:24.420598 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.420575    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf846d93-f53a-4348-b622-5a6cdd1930e2-kserve-provision-location\") pod \"bf846d93-f53a-4348-b622-5a6cdd1930e2\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") "
Sep 16 17:59:24.421030 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.420615    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swkhh\" (UniqueName: \"kubernetes.io/projected/bf846d93-f53a-4348-b622-5a6cdd1930e2-kube-api-access-swkhh\") pod \"bf846d93-f53a-4348-b622-5a6cdd1930e2\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") "
Sep 16 17:59:24.421030 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.420644    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf846d93-f53a-4348-b622-5a6cdd1930e2-proxy-tls\") pod \"bf846d93-f53a-4348-b622-5a6cdd1930e2\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") "
Sep 16 17:59:24.421030 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.420683    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf846d93-f53a-4348-b622-5a6cdd1930e2-cabundle-cert\") pod \"bf846d93-f53a-4348-b622-5a6cdd1930e2\" (UID: \"bf846d93-f53a-4348-b622-5a6cdd1930e2\") "
Sep 16 17:59:24.421030 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.420877    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf846d93-f53a-4348-b622-5a6cdd1930e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf846d93-f53a-4348-b622-5a6cdd1930e2" (UID: "bf846d93-f53a-4348-b622-5a6cdd1930e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:24.421030 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.420884    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf846d93-f53a-4348-b622-5a6cdd1930e2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bf846d93-f53a-4348-b622-5a6cdd1930e2" (UID: "bf846d93-f53a-4348-b622-5a6cdd1930e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:59:24.421191 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.421089    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf846d93-f53a-4348-b622-5a6cdd1930e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bf846d93-f53a-4348-b622-5a6cdd1930e2" (UID: "bf846d93-f53a-4348-b622-5a6cdd1930e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:59:24.421191 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.421152    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf846d93-f53a-4348-b622-5a6cdd1930e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:59:24.421191 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.421165    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf846d93-f53a-4348-b622-5a6cdd1930e2-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:59:24.422823 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.422803    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf846d93-f53a-4348-b622-5a6cdd1930e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf846d93-f53a-4348-b622-5a6cdd1930e2" (UID: "bf846d93-f53a-4348-b622-5a6cdd1930e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:59:24.422922 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.422881    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf846d93-f53a-4348-b622-5a6cdd1930e2-kube-api-access-swkhh" (OuterVolumeSpecName: "kube-api-access-swkhh") pod "bf846d93-f53a-4348-b622-5a6cdd1930e2" (UID: "bf846d93-f53a-4348-b622-5a6cdd1930e2"). InnerVolumeSpecName "kube-api-access-swkhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:59:24.522362 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.522339    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swkhh\" (UniqueName: \"kubernetes.io/projected/bf846d93-f53a-4348-b622-5a6cdd1930e2-kube-api-access-swkhh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:59:24.522362 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.522359    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf846d93-f53a-4348-b622-5a6cdd1930e2-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:59:24.522482 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.522369    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf846d93-f53a-4348-b622-5a6cdd1930e2-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:59:24.831326 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.831292    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerID="c7f24b116023b4f5af85f2cf2bc08567c5c35ec44b5fe073c366a7809515933b" exitCode=0
Sep 16 17:59:24.831508 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.831425    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" event={"ID":"bf846d93-f53a-4348-b622-5a6cdd1930e2","Type":"ContainerDied","Data":"c7f24b116023b4f5af85f2cf2bc08567c5c35ec44b5fe073c366a7809515933b"}
Sep 16 17:59:24.831508 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.831469    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"
Sep 16 17:59:24.831508 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.831484    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj" event={"ID":"bf846d93-f53a-4348-b622-5a6cdd1930e2","Type":"ContainerDied","Data":"ddec7b56ab9a7ae013651f058bb86f8cfdd3b0b2772cc8c0cb4bd684217bd1d9"}
Sep 16 17:59:24.831665 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.831523    2581 scope.go:117] "RemoveContainer" containerID="2efb59ff7f8ec4102f8b8e26f59417d02543e8cc94753552a69ae9211b23b992"
Sep 16 17:59:24.885943 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.885924    2581 scope.go:117] "RemoveContainer" containerID="c7f24b116023b4f5af85f2cf2bc08567c5c35ec44b5fe073c366a7809515933b"
Sep 16 17:59:24.892967 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.892949    2581 scope.go:117] "RemoveContainer" containerID="f38c55e75d57d18b87b733b106be19371a00e7a6630f717488b6b6e5daa5dfba"
Sep 16 17:59:24.899937 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.899920    2581 scope.go:117] "RemoveContainer" containerID="2efb59ff7f8ec4102f8b8e26f59417d02543e8cc94753552a69ae9211b23b992"
Sep 16 17:59:24.900191 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:59:24.900169    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2efb59ff7f8ec4102f8b8e26f59417d02543e8cc94753552a69ae9211b23b992\": container with ID starting with 2efb59ff7f8ec4102f8b8e26f59417d02543e8cc94753552a69ae9211b23b992 not found: ID does not exist" containerID="2efb59ff7f8ec4102f8b8e26f59417d02543e8cc94753552a69ae9211b23b992"
Sep 16 17:59:24.900255 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.900202    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2efb59ff7f8ec4102f8b8e26f59417d02543e8cc94753552a69ae9211b23b992"} err="failed to get container status \"2efb59ff7f8ec4102f8b8e26f59417d02543e8cc94753552a69ae9211b23b992\": rpc error: code = NotFound desc = could not find container \"2efb59ff7f8ec4102f8b8e26f59417d02543e8cc94753552a69ae9211b23b992\": container with ID starting with 2efb59ff7f8ec4102f8b8e26f59417d02543e8cc94753552a69ae9211b23b992 not found: ID does not exist"
Sep 16 17:59:24.900255 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.900226    2581 scope.go:117] "RemoveContainer" containerID="c7f24b116023b4f5af85f2cf2bc08567c5c35ec44b5fe073c366a7809515933b"
Sep 16 17:59:24.900499 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:59:24.900481    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f24b116023b4f5af85f2cf2bc08567c5c35ec44b5fe073c366a7809515933b\": container with ID starting with c7f24b116023b4f5af85f2cf2bc08567c5c35ec44b5fe073c366a7809515933b not found: ID does not exist" containerID="c7f24b116023b4f5af85f2cf2bc08567c5c35ec44b5fe073c366a7809515933b"
Sep 16 17:59:24.900537 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.900505    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f24b116023b4f5af85f2cf2bc08567c5c35ec44b5fe073c366a7809515933b"} err="failed to get container status \"c7f24b116023b4f5af85f2cf2bc08567c5c35ec44b5fe073c366a7809515933b\": rpc error: code = NotFound desc = could not find container \"c7f24b116023b4f5af85f2cf2bc08567c5c35ec44b5fe073c366a7809515933b\": container with ID starting with c7f24b116023b4f5af85f2cf2bc08567c5c35ec44b5fe073c366a7809515933b not found: ID does not exist"
Sep 16 17:59:24.900537 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.900520    2581 scope.go:117] "RemoveContainer" containerID="f38c55e75d57d18b87b733b106be19371a00e7a6630f717488b6b6e5daa5dfba"
Sep 16 17:59:24.900765 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:59:24.900749    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f38c55e75d57d18b87b733b106be19371a00e7a6630f717488b6b6e5daa5dfba\": container with ID starting with f38c55e75d57d18b87b733b106be19371a00e7a6630f717488b6b6e5daa5dfba not found: ID does not exist" containerID="f38c55e75d57d18b87b733b106be19371a00e7a6630f717488b6b6e5daa5dfba"
Sep 16 17:59:24.900808 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.900769    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38c55e75d57d18b87b733b106be19371a00e7a6630f717488b6b6e5daa5dfba"} err="failed to get container status \"f38c55e75d57d18b87b733b106be19371a00e7a6630f717488b6b6e5daa5dfba\": rpc error: code = NotFound desc = could not find container \"f38c55e75d57d18b87b733b106be19371a00e7a6630f717488b6b6e5daa5dfba\": container with ID starting with f38c55e75d57d18b87b733b106be19371a00e7a6630f717488b6b6e5daa5dfba not found: ID does not exist"
Sep 16 17:59:24.935457 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.935430    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"]
Sep 16 17:59:24.935659 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.935636    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kserve-container" containerID="cri-o://8659c528736464a2068eb98a43ba71edf564edd14e4079df6662fd14f03a667c" gracePeriod=30
Sep 16 17:59:24.935731 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.935643    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kube-rbac-proxy" containerID="cri-o://2860d61b913dc1e33e3728528fe359be37b7f9a51687d2c759c5d5cb52408863" gracePeriod=30
Sep 16 17:59:24.937079 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.937061    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"]
Sep 16 17:59:24.940644 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.940626    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-sj2rj"]
Sep 16 17:59:24.957155 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:24.957108    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:25.765995 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:25.765970    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" path="/var/lib/kubelet/pods/bf846d93-f53a-4348-b622-5a6cdd1930e2/volumes"
Sep 16 17:59:25.836489 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:25.836463    2581 generic.go:356] "Generic (PLEG): container finished" podID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerID="3491bfb61bfdaf9f46683c1c8044f55f890f6194c50638c58fe02329188dfb11" exitCode=0
Sep 16 17:59:25.836592 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:25.836539    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" event={"ID":"03fb78d7-5343-4534-8f6f-273a6221a8d1","Type":"ContainerDied","Data":"3491bfb61bfdaf9f46683c1c8044f55f890f6194c50638c58fe02329188dfb11"}
Sep 16 17:59:25.839583 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:25.839562    2581 generic.go:356] "Generic (PLEG): container finished" podID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerID="2860d61b913dc1e33e3728528fe359be37b7f9a51687d2c759c5d5cb52408863" exitCode=2
Sep 16 17:59:25.839662 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:25.839594    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" event={"ID":"a0aa9912-225e-47f9-a3f1-b89d3a4edf17","Type":"ContainerDied","Data":"2860d61b913dc1e33e3728528fe359be37b7f9a51687d2c759c5d5cb52408863"}
Sep 16 17:59:26.847377 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:26.847350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" event={"ID":"03fb78d7-5343-4534-8f6f-273a6221a8d1","Type":"ContainerStarted","Data":"45aee97a80ce90d0c1e0dfd43b877094eec1ee38220e22d9caf14aac7e471228"}
Sep 16 17:59:26.847775 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:26.847405    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" event={"ID":"03fb78d7-5343-4534-8f6f-273a6221a8d1","Type":"ContainerStarted","Data":"2597e87a3a3429cff152dc5e187b4acf052e6cfff7897e5683812a630cd110b1"}
Sep 16 17:59:26.847775 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:26.847579    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:27.852227 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:27.852196    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:27.853304 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:27.853278    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 17:59:28.857706 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:28.857671    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 17:59:29.689833 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:29.689796    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 16 17:59:30.009390 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:30.009316    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:30.146298 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:30.146250    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:31.832694 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:31.832661    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:59:31.835015 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:31.834997    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 17:59:33.861853 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:33.861828    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 17:59:33.862285 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:33.862254    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 17:59:33.900193 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:33.900141    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" podStartSLOduration=10.085640748 podStartE2EDuration="13.900127539s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.814486791s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:59:26.881036648 +0000 UTC m=+2095.669659192" watchObservedRunningTime="2026-09-16 17:59:33.900127539 +0000 UTC m=+2102.688750072"
Sep 16 17:59:34.689584 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:34.689550    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 16 17:59:39.689128 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:39.689089    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 16 17:59:39.689571 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:39.689220    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:59:43.862858 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:43.862820    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 17:59:44.689831 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:44.689792    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 16 17:59:49.688855 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:49.688817    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 16 17:59:53.862592 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:53.862561    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 17:59:54.689404 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:54.689372    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 16 17:59:55.579223 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.579200    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:59:55.642108 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.642085    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjkjk\" (UniqueName: \"kubernetes.io/projected/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kube-api-access-hjkjk\") pod \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") "
Sep 16 17:59:55.642217 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.642126    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kserve-provision-location\") pod \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") "
Sep 16 17:59:55.642217 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.642163    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-cabundle-cert\") pod \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") "
Sep 16 17:59:55.642334 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.642252    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-proxy-tls\") pod \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") "
Sep 16 17:59:55.642334 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.642289    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kube-rbac-proxy-sar-config\") pod \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\" (UID: \"a0aa9912-225e-47f9-a3f1-b89d3a4edf17\") "
Sep 16 17:59:55.642525 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.642490    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a0aa9912-225e-47f9-a3f1-b89d3a4edf17" (UID: "a0aa9912-225e-47f9-a3f1-b89d3a4edf17"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:59:55.642706 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.642683    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a0aa9912-225e-47f9-a3f1-b89d3a4edf17" (UID: "a0aa9912-225e-47f9-a3f1-b89d3a4edf17"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:59:55.644320 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.644300    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kube-api-access-hjkjk" (OuterVolumeSpecName: "kube-api-access-hjkjk") pod "a0aa9912-225e-47f9-a3f1-b89d3a4edf17" (UID: "a0aa9912-225e-47f9-a3f1-b89d3a4edf17"). InnerVolumeSpecName "kube-api-access-hjkjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:59:55.644479 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.644458    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0aa9912-225e-47f9-a3f1-b89d3a4edf17" (UID: "a0aa9912-225e-47f9-a3f1-b89d3a4edf17"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:59:55.654507 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.654484    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a0aa9912-225e-47f9-a3f1-b89d3a4edf17" (UID: "a0aa9912-225e-47f9-a3f1-b89d3a4edf17"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:55.743332 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.743280    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:59:55.743332 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.743305    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:59:55.743332 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.743315    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjkjk\" (UniqueName: \"kubernetes.io/projected/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kube-api-access-hjkjk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:59:55.743332 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.743325    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:59:55.743332 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.743333    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0aa9912-225e-47f9-a3f1-b89d3a4edf17-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 17:59:55.976158 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.976131    2581 generic.go:356] "Generic (PLEG): container finished" podID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerID="8659c528736464a2068eb98a43ba71edf564edd14e4079df6662fd14f03a667c" exitCode=137
Sep 16 17:59:55.976265 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.976190    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"
Sep 16 17:59:55.976265 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.976196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" event={"ID":"a0aa9912-225e-47f9-a3f1-b89d3a4edf17","Type":"ContainerDied","Data":"8659c528736464a2068eb98a43ba71edf564edd14e4079df6662fd14f03a667c"}
Sep 16 17:59:55.976265 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.976250    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md" event={"ID":"a0aa9912-225e-47f9-a3f1-b89d3a4edf17","Type":"ContainerDied","Data":"1f873a1ded6dbf9b10bf6c57f3f972a04cd6b3cd63f8cb6e8ffc62ea2ffcfb6b"}
Sep 16 17:59:55.976425 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.976288    2581 scope.go:117] "RemoveContainer" containerID="2860d61b913dc1e33e3728528fe359be37b7f9a51687d2c759c5d5cb52408863"
Sep 16 17:59:55.984596 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.984581    2581 scope.go:117] "RemoveContainer" containerID="8659c528736464a2068eb98a43ba71edf564edd14e4079df6662fd14f03a667c"
Sep 16 17:59:55.991564 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.991546    2581 scope.go:117] "RemoveContainer" containerID="f598e834d87c38bf8476fcb3b050d2c231e790b25a668718f66d1f31d9abe371"
Sep 16 17:59:55.998505 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.998457    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"]
Sep 16 17:59:55.998801 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.998777    2581 scope.go:117] "RemoveContainer" containerID="2860d61b913dc1e33e3728528fe359be37b7f9a51687d2c759c5d5cb52408863"
Sep 16 17:59:55.999097 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:59:55.999079    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2860d61b913dc1e33e3728528fe359be37b7f9a51687d2c759c5d5cb52408863\": container with ID starting with 2860d61b913dc1e33e3728528fe359be37b7f9a51687d2c759c5d5cb52408863 not found: ID does not exist" containerID="2860d61b913dc1e33e3728528fe359be37b7f9a51687d2c759c5d5cb52408863"
Sep 16 17:59:55.999165 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.999106    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2860d61b913dc1e33e3728528fe359be37b7f9a51687d2c759c5d5cb52408863"} err="failed to get container status \"2860d61b913dc1e33e3728528fe359be37b7f9a51687d2c759c5d5cb52408863\": rpc error: code = NotFound desc = could not find container \"2860d61b913dc1e33e3728528fe359be37b7f9a51687d2c759c5d5cb52408863\": container with ID starting with 2860d61b913dc1e33e3728528fe359be37b7f9a51687d2c759c5d5cb52408863 not found: ID does not exist"
Sep 16 17:59:55.999165 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.999123    2581 scope.go:117] "RemoveContainer" containerID="8659c528736464a2068eb98a43ba71edf564edd14e4079df6662fd14f03a667c"
Sep 16 17:59:55.999380 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:59:55.999362    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8659c528736464a2068eb98a43ba71edf564edd14e4079df6662fd14f03a667c\": container with ID starting with 8659c528736464a2068eb98a43ba71edf564edd14e4079df6662fd14f03a667c not found: ID does not exist" containerID="8659c528736464a2068eb98a43ba71edf564edd14e4079df6662fd14f03a667c"
Sep 16 17:59:55.999424 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.999388    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8659c528736464a2068eb98a43ba71edf564edd14e4079df6662fd14f03a667c"} err="failed to get container status \"8659c528736464a2068eb98a43ba71edf564edd14e4079df6662fd14f03a667c\": rpc error: code = NotFound desc = could not find container \"8659c528736464a2068eb98a43ba71edf564edd14e4079df6662fd14f03a667c\": container with ID starting with 8659c528736464a2068eb98a43ba71edf564edd14e4079df6662fd14f03a667c not found: ID does not exist"
Sep 16 17:59:55.999424 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.999404    2581 scope.go:117] "RemoveContainer" containerID="f598e834d87c38bf8476fcb3b050d2c231e790b25a668718f66d1f31d9abe371"
Sep 16 17:59:55.999649 ip-10-0-128-31 kubenswrapper[2581]: E0916 17:59:55.999631    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f598e834d87c38bf8476fcb3b050d2c231e790b25a668718f66d1f31d9abe371\": container with ID starting with f598e834d87c38bf8476fcb3b050d2c231e790b25a668718f66d1f31d9abe371 not found: ID does not exist" containerID="f598e834d87c38bf8476fcb3b050d2c231e790b25a668718f66d1f31d9abe371"
Sep 16 17:59:55.999705 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:55.999658    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f598e834d87c38bf8476fcb3b050d2c231e790b25a668718f66d1f31d9abe371"} err="failed to get container status \"f598e834d87c38bf8476fcb3b050d2c231e790b25a668718f66d1f31d9abe371\": rpc error: code = NotFound desc = could not find container \"f598e834d87c38bf8476fcb3b050d2c231e790b25a668718f66d1f31d9abe371\": container with ID starting with f598e834d87c38bf8476fcb3b050d2c231e790b25a668718f66d1f31d9abe371 not found: ID does not exist"
Sep 16 17:59:56.004260 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:56.004241    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-8q2md"]
Sep 16 17:59:57.747039 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:57.747005    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:57.766267 ip-10-0-128-31 kubenswrapper[2581]: I0916 17:59:57.766239    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" path="/var/lib/kubelet/pods/a0aa9912-225e-47f9-a3f1-b89d3a4edf17/volumes"
Sep 16 18:00:03.862973 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:03.862935    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 18:00:13.862739 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:13.862698    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 18:00:18.723682 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:18.723647    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:18.780744 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:18.780719    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:23.863217 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:23.863131    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 18:00:33.634210 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:33.634176    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:33.863680 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:33.863657    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 18:00:40.345951 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:40.345919    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"]
Sep 16 18:00:40.346340 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:40.346158    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kserve-container" containerID="cri-o://2597e87a3a3429cff152dc5e187b4acf052e6cfff7897e5683812a630cd110b1" gracePeriod=30
Sep 16 18:00:40.346340 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:40.346222    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kube-rbac-proxy" containerID="cri-o://45aee97a80ce90d0c1e0dfd43b877094eec1ee38220e22d9caf14aac7e471228" gracePeriod=30
Sep 16 18:00:40.360753 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:40.360733    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:41.151117 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:41.151090    2581 generic.go:356] "Generic (PLEG): container finished" podID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerID="45aee97a80ce90d0c1e0dfd43b877094eec1ee38220e22d9caf14aac7e471228" exitCode=2
Sep 16 18:00:41.151273 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:41.151157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" event={"ID":"03fb78d7-5343-4534-8f6f-273a6221a8d1","Type":"ContainerDied","Data":"45aee97a80ce90d0c1e0dfd43b877094eec1ee38220e22d9caf14aac7e471228"}
Sep 16 18:00:43.858251 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:43.858208    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Sep 16 18:00:43.862546 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:43.862515    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 18:00:44.166140 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.166112    2581 generic.go:356] "Generic (PLEG): container finished" podID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerID="2597e87a3a3429cff152dc5e187b4acf052e6cfff7897e5683812a630cd110b1" exitCode=0
Sep 16 18:00:44.166266 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.166176    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" event={"ID":"03fb78d7-5343-4534-8f6f-273a6221a8d1","Type":"ContainerDied","Data":"2597e87a3a3429cff152dc5e187b4acf052e6cfff7897e5683812a630cd110b1"}
Sep 16 18:00:44.185076 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.185058    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 18:00:44.282870 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.282844    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03fb78d7-5343-4534-8f6f-273a6221a8d1-proxy-tls\") pod \"03fb78d7-5343-4534-8f6f-273a6221a8d1\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") "
Sep 16 18:00:44.283010 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.282876    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnnrm\" (UniqueName: \"kubernetes.io/projected/03fb78d7-5343-4534-8f6f-273a6221a8d1-kube-api-access-rnnrm\") pod \"03fb78d7-5343-4534-8f6f-273a6221a8d1\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") "
Sep 16 18:00:44.283010 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.282918    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03fb78d7-5343-4534-8f6f-273a6221a8d1-kserve-provision-location\") pod \"03fb78d7-5343-4534-8f6f-273a6221a8d1\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") "
Sep 16 18:00:44.283131 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.283024    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03fb78d7-5343-4534-8f6f-273a6221a8d1-kube-rbac-proxy-sar-config\") pod \"03fb78d7-5343-4534-8f6f-273a6221a8d1\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") "
Sep 16 18:00:44.283131 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.283066    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03fb78d7-5343-4534-8f6f-273a6221a8d1-cabundle-cert\") pod \"03fb78d7-5343-4534-8f6f-273a6221a8d1\" (UID: \"03fb78d7-5343-4534-8f6f-273a6221a8d1\") "
Sep 16 18:00:44.283286 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.283201    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03fb78d7-5343-4534-8f6f-273a6221a8d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "03fb78d7-5343-4534-8f6f-273a6221a8d1" (UID: "03fb78d7-5343-4534-8f6f-273a6221a8d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:00:44.283357 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.283331    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03fb78d7-5343-4534-8f6f-273a6221a8d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "03fb78d7-5343-4534-8f6f-273a6221a8d1" (UID: "03fb78d7-5343-4534-8f6f-273a6221a8d1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:00:44.283410 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.283362    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03fb78d7-5343-4534-8f6f-273a6221a8d1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "03fb78d7-5343-4534-8f6f-273a6221a8d1" (UID: "03fb78d7-5343-4534-8f6f-273a6221a8d1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:00:44.285042 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.285026    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03fb78d7-5343-4534-8f6f-273a6221a8d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03fb78d7-5343-4534-8f6f-273a6221a8d1" (UID: "03fb78d7-5343-4534-8f6f-273a6221a8d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:00:44.285125 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.285088    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03fb78d7-5343-4534-8f6f-273a6221a8d1-kube-api-access-rnnrm" (OuterVolumeSpecName: "kube-api-access-rnnrm") pod "03fb78d7-5343-4534-8f6f-273a6221a8d1" (UID: "03fb78d7-5343-4534-8f6f-273a6221a8d1"). InnerVolumeSpecName "kube-api-access-rnnrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:00:44.383745 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.383688    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03fb78d7-5343-4534-8f6f-273a6221a8d1-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:00:44.383745 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.383712    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnnrm\" (UniqueName: \"kubernetes.io/projected/03fb78d7-5343-4534-8f6f-273a6221a8d1-kube-api-access-rnnrm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:00:44.383745 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.383725    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03fb78d7-5343-4534-8f6f-273a6221a8d1-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:00:44.383745 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.383737    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03fb78d7-5343-4534-8f6f-273a6221a8d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:00:44.383966 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:44.383751    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03fb78d7-5343-4534-8f6f-273a6221a8d1-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:00:45.175301 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.175277    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"
Sep 16 18:00:45.175661 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.175306    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s" event={"ID":"03fb78d7-5343-4534-8f6f-273a6221a8d1","Type":"ContainerDied","Data":"106f4547fae45f2d458a488efbe0a0e909057ea5075405044ed0a38a72509c59"}
Sep 16 18:00:45.175661 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.175373    2581 scope.go:117] "RemoveContainer" containerID="45aee97a80ce90d0c1e0dfd43b877094eec1ee38220e22d9caf14aac7e471228"
Sep 16 18:00:45.184101 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.184085    2581 scope.go:117] "RemoveContainer" containerID="2597e87a3a3429cff152dc5e187b4acf052e6cfff7897e5683812a630cd110b1"
Sep 16 18:00:45.191176 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.191159    2581 scope.go:117] "RemoveContainer" containerID="3491bfb61bfdaf9f46683c1c8044f55f890f6194c50638c58fe02329188dfb11"
Sep 16 18:00:45.199709 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.199686    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"]
Sep 16 18:00:45.206755 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.206736    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-fr64s"]
Sep 16 18:00:45.544495 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.544469    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:45.609514 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.609486    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"]
Sep 16 18:00:45.609991 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.609975    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="storage-initializer"
Sep 16 18:00:45.610080 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.609994    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="storage-initializer"
Sep 16 18:00:45.610080 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610010    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kube-rbac-proxy"
Sep 16 18:00:45.610080 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610018    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kube-rbac-proxy"
Sep 16 18:00:45.610080 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610029    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="storage-initializer"
Sep 16 18:00:45.610080 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610036    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="storage-initializer"
Sep 16 18:00:45.610080 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610050    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kserve-container"
Sep 16 18:00:45.610080 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610058    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kserve-container"
Sep 16 18:00:45.610080 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610069    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kserve-container"
Sep 16 18:00:45.610080 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610077    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kserve-container"
Sep 16 18:00:45.610374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610089    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kube-rbac-proxy"
Sep 16 18:00:45.610374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610097    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kube-rbac-proxy"
Sep 16 18:00:45.610374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610106    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kserve-container"
Sep 16 18:00:45.610374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610114    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kserve-container"
Sep 16 18:00:45.610374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610127    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="storage-initializer"
Sep 16 18:00:45.610374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610136    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="storage-initializer"
Sep 16 18:00:45.610374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610146    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kube-rbac-proxy"
Sep 16 18:00:45.610374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610154    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kube-rbac-proxy"
Sep 16 18:00:45.610374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610226    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kserve-container"
Sep 16 18:00:45.610374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610237    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kube-rbac-proxy"
Sep 16 18:00:45.610374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610248    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kserve-container"
Sep 16 18:00:45.610374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610257    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0aa9912-225e-47f9-a3f1-b89d3a4edf17" containerName="kube-rbac-proxy"
Sep 16 18:00:45.610374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610267    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" containerName="kube-rbac-proxy"
Sep 16 18:00:45.610374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.610274    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf846d93-f53a-4348-b622-5a6cdd1930e2" containerName="kserve-container"
Sep 16 18:00:45.615036 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.615019    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.617986 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.617962    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 16 18:00:45.618099 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.617962    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 18:00:45.618099 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.617962    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:00:45.618309 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.618292    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:00:45.618399 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.618354    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:00:45.618508 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.618433    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:00:45.618508 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.618477    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\""
Sep 16 18:00:45.624383 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.624361    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"]
Sep 16 18:00:45.627214 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.627192    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:45.650457 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.650438    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"]
Sep 16 18:00:45.654295 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.654279    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.656818 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.656803    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 18:00:45.664527 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.664506    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"]
Sep 16 18:00:45.694401 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.694381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.694509 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.694412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.694509 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.694433    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.694509 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.694484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv7v9\" (UniqueName: \"kubernetes.io/projected/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kube-api-access-mv7v9\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.694657 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.694521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.765406 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.765384    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03fb78d7-5343-4534-8f6f-273a6221a8d1" path="/var/lib/kubelet/pods/03fb78d7-5343-4534-8f6f-273a6221a8d1/volumes"
Sep 16 18:00:45.795778 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.795732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-bdkx2\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.795778 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.795760    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg5f4\" (UniqueName: \"kubernetes.io/projected/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kube-api-access-dg5f4\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-bdkx2\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.795778 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.795786    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-bdkx2\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.795967 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.795872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.795967 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.795913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.795967 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.795937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.795967 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.795963    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv7v9\" (UniqueName: \"kubernetes.io/projected/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kube-api-access-mv7v9\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.796151 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.795988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.796151 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.796029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-bdkx2\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.796254 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.796146    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-bdkx2\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.796307 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.796292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.796610 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.796595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.796648 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.796622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.798358 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.798342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.805131 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.805113    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv7v9\" (UniqueName: \"kubernetes.io/projected/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kube-api-access-mv7v9\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.897182 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.897153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-bdkx2\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.897280 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.897190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-bdkx2\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.897280 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.897214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-bdkx2\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.897280 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.897230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg5f4\" (UniqueName: \"kubernetes.io/projected/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kube-api-access-dg5f4\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-bdkx2\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.897280 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.897249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-bdkx2\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.897510 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.897493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-bdkx2\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.897883 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.897858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-bdkx2\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.898001 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.897858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-bdkx2\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.899421 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.899398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-bdkx2\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.908432 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.908415    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg5f4\" (UniqueName: \"kubernetes.io/projected/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kube-api-access-dg5f4\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-bdkx2\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:45.926883 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.926861    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:45.965069 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:45.965040    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:46.202774 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:46.202751    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"]
Sep 16 18:00:46.204463 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:00:46.204423    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732c0b9d_bd3b_4cbc_8f16_b2fa0dbf09ff.slice/crio-5627eec470ad25a0af9ec4862e906f562f403f39d2e4dc6b69fbeb56166c7aee WatchSource:0}: Error finding container 5627eec470ad25a0af9ec4862e906f562f403f39d2e4dc6b69fbeb56166c7aee: Status 404 returned error can't find the container with id 5627eec470ad25a0af9ec4862e906f562f403f39d2e4dc6b69fbeb56166c7aee
Sep 16 18:00:46.245660 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:46.245633    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"]
Sep 16 18:00:46.246688 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:00:46.246654    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257a0fe8_8cb2_4875_9fcb_9019c4f2c63e.slice/crio-3eab5bb4a46818e333c0dd9c013b20f4473cd97f262a18861ca962f003dfbe84 WatchSource:0}: Error finding container 3eab5bb4a46818e333c0dd9c013b20f4473cd97f262a18861ca962f003dfbe84: Status 404 returned error can't find the container with id 3eab5bb4a46818e333c0dd9c013b20f4473cd97f262a18861ca962f003dfbe84
Sep 16 18:00:46.415256 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:46.415220    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:46.471302 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:46.471222    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:47.186231 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:47.186198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" event={"ID":"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff","Type":"ContainerStarted","Data":"0c16011fa6ae1c316f7bd0528622ece029609fad9b9f5a2c100e995002aa5fce"}
Sep 16 18:00:47.186401 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:47.186270    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" event={"ID":"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff","Type":"ContainerStarted","Data":"5627eec470ad25a0af9ec4862e906f562f403f39d2e4dc6b69fbeb56166c7aee"}
Sep 16 18:00:47.187699 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:47.187674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" event={"ID":"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e","Type":"ContainerStarted","Data":"a4ca18cc8861b87364ee9d1a4e86253bc32c1a04f047a45e0b223cd1a3d22197"}
Sep 16 18:00:47.187803 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:47.187720    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" event={"ID":"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e","Type":"ContainerStarted","Data":"3eab5bb4a46818e333c0dd9c013b20f4473cd97f262a18861ca962f003dfbe84"}
Sep 16 18:00:50.202366 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:50.202336    2581 generic.go:356] "Generic (PLEG): container finished" podID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerID="0c16011fa6ae1c316f7bd0528622ece029609fad9b9f5a2c100e995002aa5fce" exitCode=0
Sep 16 18:00:50.202754 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:50.202413    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" event={"ID":"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff","Type":"ContainerDied","Data":"0c16011fa6ae1c316f7bd0528622ece029609fad9b9f5a2c100e995002aa5fce"}
Sep 16 18:00:51.208109 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:51.208077    2581 generic.go:356] "Generic (PLEG): container finished" podID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerID="a4ca18cc8861b87364ee9d1a4e86253bc32c1a04f047a45e0b223cd1a3d22197" exitCode=0
Sep 16 18:00:51.208473 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:51.208151    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" event={"ID":"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e","Type":"ContainerDied","Data":"a4ca18cc8861b87364ee9d1a4e86253bc32c1a04f047a45e0b223cd1a3d22197"}
Sep 16 18:00:51.210347 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:51.210327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" event={"ID":"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff","Type":"ContainerStarted","Data":"1ca9ac0d799396fcf793234bf811a0526a7d0fe1cab0a4e76e7e078df653cb69"}
Sep 16 18:00:51.210444 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:51.210372    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" event={"ID":"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff","Type":"ContainerStarted","Data":"3dd0cae3845e49098cf7d27a75705470117063622cc3a28793a8d0a5eb079689"}
Sep 16 18:00:51.210668 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:51.210635    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:51.210736 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:51.210695    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:51.211783 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:51.211757    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 18:00:51.249900 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:51.249848    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" podStartSLOduration=2.532139158 podStartE2EDuration="6.249835159s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.717696001s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:00:51.248084884 +0000 UTC m=+2180.036707414" watchObservedRunningTime="2026-09-16 18:00:51.249835159 +0000 UTC m=+2180.038457692"
Sep 16 18:00:52.217821 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:52.217785    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" event={"ID":"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e","Type":"ContainerStarted","Data":"aaa81755fc47d48aedc72034142f1b2daa89439ece0933e0667ac27af2cc80be"}
Sep 16 18:00:52.218203 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:52.217847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" event={"ID":"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e","Type":"ContainerStarted","Data":"87526bffa6c15fa2e46254e5ad60df2cf86b81a1390250e4b727768c1b388c1a"}
Sep 16 18:00:52.218203 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:52.218056    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 18:00:52.218290 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:52.218272    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:52.245584 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:52.245541    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" podStartSLOduration=3.282072128 podStartE2EDuration="7.245528585s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.963456457s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:00:52.244410596 +0000 UTC m=+2181.033033130" watchObservedRunningTime="2026-09-16 18:00:52.245528585 +0000 UTC m=+2181.034151145"
Sep 16 18:00:53.222942 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:53.222913    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:53.223984 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:53.223955    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 18:00:54.227330 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:54.227289    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 18:00:57.222440 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:57.222415    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:00:57.222867 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:57.222844    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 18:00:59.231486 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:59.231457    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:00:59.231982 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:00:59.231952    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 18:01:07.223455 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:07.223412    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 18:01:09.232230 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:09.232193    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 18:01:17.223374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:17.223339    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 18:01:19.231851 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:19.231817    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 18:01:27.223133 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:27.223088    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 18:01:29.235344 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:29.235298    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 18:01:37.223469 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:37.223429    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 18:01:39.231872 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:39.231832    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 18:01:42.820465 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:42.820434    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:01:47.222795 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:47.222755    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 18:01:47.978215 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:47.978181    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:01:48.412167 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:48.412135    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:01:49.231863 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:49.231828    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 18:01:57.223973 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:57.223883    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:01:57.437699 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:57.437670    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"]
Sep 16 18:01:57.437925 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:57.437866    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kserve-container" containerID="cri-o://87526bffa6c15fa2e46254e5ad60df2cf86b81a1390250e4b727768c1b388c1a" gracePeriod=30
Sep 16 18:01:57.438003 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:57.437953    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kube-rbac-proxy" containerID="cri-o://aaa81755fc47d48aedc72034142f1b2daa89439ece0933e0667ac27af2cc80be" gracePeriod=30
Sep 16 18:01:57.438169 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:57.438124    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 18:01:58.511688 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:58.511658    2581 generic.go:356] "Generic (PLEG): container finished" podID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerID="aaa81755fc47d48aedc72034142f1b2daa89439ece0933e0667ac27af2cc80be" exitCode=2
Sep 16 18:01:58.512086 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:58.511736    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" event={"ID":"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e","Type":"ContainerDied","Data":"aaa81755fc47d48aedc72034142f1b2daa89439ece0933e0667ac27af2cc80be"}
Sep 16 18:01:59.227850 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:01:59.227808    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 16 18:02:01.075769 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.075739    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:02:01.137671 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.137602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kube-rbac-proxy-sar-config\") pod \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") "
Sep 16 18:02:01.137671 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.137651    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-proxy-tls\") pod \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") "
Sep 16 18:02:01.137857 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.137683    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-cabundle-cert\") pod \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") "
Sep 16 18:02:01.137857 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.137753    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg5f4\" (UniqueName: \"kubernetes.io/projected/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kube-api-access-dg5f4\") pod \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") "
Sep 16 18:02:01.137857 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.137774    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kserve-provision-location\") pod \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\" (UID: \"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e\") "
Sep 16 18:02:01.138097 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.138066    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" (UID: "257a0fe8-8cb2-4875-9fcb-9019c4f2c63e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:02:01.138162 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.138141    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" (UID: "257a0fe8-8cb2-4875-9fcb-9019c4f2c63e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:02:01.138204 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.138144    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" (UID: "257a0fe8-8cb2-4875-9fcb-9019c4f2c63e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:02:01.139698 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.139667    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" (UID: "257a0fe8-8cb2-4875-9fcb-9019c4f2c63e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:02:01.140087 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.140069    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kube-api-access-dg5f4" (OuterVolumeSpecName: "kube-api-access-dg5f4") pod "257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" (UID: "257a0fe8-8cb2-4875-9fcb-9019c4f2c63e"). InnerVolumeSpecName "kube-api-access-dg5f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:02:01.238548 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.238528    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg5f4\" (UniqueName: \"kubernetes.io/projected/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kube-api-access-dg5f4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:02:01.238548 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.238549    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:02:01.238662 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.238558    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:02:01.238662 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.238566    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:02:01.238662 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.238575    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:02:01.525167 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.525143    2581 generic.go:356] "Generic (PLEG): container finished" podID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerID="87526bffa6c15fa2e46254e5ad60df2cf86b81a1390250e4b727768c1b388c1a" exitCode=0
Sep 16 18:02:01.525262 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.525201    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"
Sep 16 18:02:01.525308 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.525201    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" event={"ID":"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e","Type":"ContainerDied","Data":"87526bffa6c15fa2e46254e5ad60df2cf86b81a1390250e4b727768c1b388c1a"}
Sep 16 18:02:01.525349 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.525322    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2" event={"ID":"257a0fe8-8cb2-4875-9fcb-9019c4f2c63e","Type":"ContainerDied","Data":"3eab5bb4a46818e333c0dd9c013b20f4473cd97f262a18861ca962f003dfbe84"}
Sep 16 18:02:01.525349 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.525347    2581 scope.go:117] "RemoveContainer" containerID="aaa81755fc47d48aedc72034142f1b2daa89439ece0933e0667ac27af2cc80be"
Sep 16 18:02:01.533803 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.533786    2581 scope.go:117] "RemoveContainer" containerID="87526bffa6c15fa2e46254e5ad60df2cf86b81a1390250e4b727768c1b388c1a"
Sep 16 18:02:01.540958 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.540939    2581 scope.go:117] "RemoveContainer" containerID="a4ca18cc8861b87364ee9d1a4e86253bc32c1a04f047a45e0b223cd1a3d22197"
Sep 16 18:02:01.548164 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.548146    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"]
Sep 16 18:02:01.548708 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.548691    2581 scope.go:117] "RemoveContainer" containerID="aaa81755fc47d48aedc72034142f1b2daa89439ece0933e0667ac27af2cc80be"
Sep 16 18:02:01.548985 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:02:01.548966    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa81755fc47d48aedc72034142f1b2daa89439ece0933e0667ac27af2cc80be\": container with ID starting with aaa81755fc47d48aedc72034142f1b2daa89439ece0933e0667ac27af2cc80be not found: ID does not exist" containerID="aaa81755fc47d48aedc72034142f1b2daa89439ece0933e0667ac27af2cc80be"
Sep 16 18:02:01.549057 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.548997    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa81755fc47d48aedc72034142f1b2daa89439ece0933e0667ac27af2cc80be"} err="failed to get container status \"aaa81755fc47d48aedc72034142f1b2daa89439ece0933e0667ac27af2cc80be\": rpc error: code = NotFound desc = could not find container \"aaa81755fc47d48aedc72034142f1b2daa89439ece0933e0667ac27af2cc80be\": container with ID starting with aaa81755fc47d48aedc72034142f1b2daa89439ece0933e0667ac27af2cc80be not found: ID does not exist"
Sep 16 18:02:01.549057 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.549018    2581 scope.go:117] "RemoveContainer" containerID="87526bffa6c15fa2e46254e5ad60df2cf86b81a1390250e4b727768c1b388c1a"
Sep 16 18:02:01.549299 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:02:01.549277    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87526bffa6c15fa2e46254e5ad60df2cf86b81a1390250e4b727768c1b388c1a\": container with ID starting with 87526bffa6c15fa2e46254e5ad60df2cf86b81a1390250e4b727768c1b388c1a not found: ID does not exist" containerID="87526bffa6c15fa2e46254e5ad60df2cf86b81a1390250e4b727768c1b388c1a"
Sep 16 18:02:01.549401 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.549301    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87526bffa6c15fa2e46254e5ad60df2cf86b81a1390250e4b727768c1b388c1a"} err="failed to get container status \"87526bffa6c15fa2e46254e5ad60df2cf86b81a1390250e4b727768c1b388c1a\": rpc error: code = NotFound desc = could not find container \"87526bffa6c15fa2e46254e5ad60df2cf86b81a1390250e4b727768c1b388c1a\": container with ID starting with 87526bffa6c15fa2e46254e5ad60df2cf86b81a1390250e4b727768c1b388c1a not found: ID does not exist"
Sep 16 18:02:01.549401 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.549315    2581 scope.go:117] "RemoveContainer" containerID="a4ca18cc8861b87364ee9d1a4e86253bc32c1a04f047a45e0b223cd1a3d22197"
Sep 16 18:02:01.549665 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:02:01.549636    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4ca18cc8861b87364ee9d1a4e86253bc32c1a04f047a45e0b223cd1a3d22197\": container with ID starting with a4ca18cc8861b87364ee9d1a4e86253bc32c1a04f047a45e0b223cd1a3d22197 not found: ID does not exist" containerID="a4ca18cc8861b87364ee9d1a4e86253bc32c1a04f047a45e0b223cd1a3d22197"
Sep 16 18:02:01.549766 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.549672    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ca18cc8861b87364ee9d1a4e86253bc32c1a04f047a45e0b223cd1a3d22197"} err="failed to get container status \"a4ca18cc8861b87364ee9d1a4e86253bc32c1a04f047a45e0b223cd1a3d22197\": rpc error: code = NotFound desc = could not find container \"a4ca18cc8861b87364ee9d1a4e86253bc32c1a04f047a45e0b223cd1a3d22197\": container with ID starting with a4ca18cc8861b87364ee9d1a4e86253bc32c1a04f047a45e0b223cd1a3d22197 not found: ID does not exist"
Sep 16 18:02:01.551541 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.551520    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-bdkx2"]
Sep 16 18:02:01.766304 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:01.766273    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" path="/var/lib/kubelet/pods/257a0fe8-8cb2-4875-9fcb-9019c4f2c63e/volumes"
Sep 16 18:02:55.714198 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:55.714164    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"]
Sep 16 18:02:55.714642 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:55.714416    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kserve-container" containerID="cri-o://3dd0cae3845e49098cf7d27a75705470117063622cc3a28793a8d0a5eb079689" gracePeriod=30
Sep 16 18:02:55.714642 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:55.714424    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kube-rbac-proxy" containerID="cri-o://1ca9ac0d799396fcf793234bf811a0526a7d0fe1cab0a4e76e7e078df653cb69" gracePeriod=30
Sep 16 18:02:55.776034 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:55.776007    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:02:56.046476 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.046449    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"]
Sep 16 18:02:56.046752 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.046741    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kserve-container"
Sep 16 18:02:56.046796 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.046756    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kserve-container"
Sep 16 18:02:56.046796 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.046768    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kube-rbac-proxy"
Sep 16 18:02:56.046796 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.046773    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kube-rbac-proxy"
Sep 16 18:02:56.046796 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.046783    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="storage-initializer"
Sep 16 18:02:56.046796 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.046788    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="storage-initializer"
Sep 16 18:02:56.046988 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.046843    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kserve-container"
Sep 16 18:02:56.046988 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.046850    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="257a0fe8-8cb2-4875-9fcb-9019c4f2c63e" containerName="kube-rbac-proxy"
Sep 16 18:02:56.050066 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.050051    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.059318 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.059300    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 18:02:56.096595 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.096574    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"]
Sep 16 18:02:56.213847 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.213826    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cntkr\" (UniqueName: \"kubernetes.io/projected/818162f8-f176-44b1-9fc0-2ac622116aa1-kube-api-access-cntkr\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-xckzd\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.213847 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.213855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/818162f8-f176-44b1-9fc0-2ac622116aa1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-xckzd\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.214050 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.213882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/818162f8-f176-44b1-9fc0-2ac622116aa1-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-xckzd\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.214050 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.213965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/818162f8-f176-44b1-9fc0-2ac622116aa1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-xckzd\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.214050 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.214004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/818162f8-f176-44b1-9fc0-2ac622116aa1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-xckzd\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.315212 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.315156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/818162f8-f176-44b1-9fc0-2ac622116aa1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-xckzd\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.315212 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.315194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cntkr\" (UniqueName: \"kubernetes.io/projected/818162f8-f176-44b1-9fc0-2ac622116aa1-kube-api-access-cntkr\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-xckzd\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.315335 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.315214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/818162f8-f176-44b1-9fc0-2ac622116aa1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-xckzd\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.315335 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.315241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/818162f8-f176-44b1-9fc0-2ac622116aa1-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-xckzd\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.315335 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.315263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/818162f8-f176-44b1-9fc0-2ac622116aa1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-xckzd\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.315564 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.315541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/818162f8-f176-44b1-9fc0-2ac622116aa1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-xckzd\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.315885 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.315864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/818162f8-f176-44b1-9fc0-2ac622116aa1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-xckzd\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.316008 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.315991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/818162f8-f176-44b1-9fc0-2ac622116aa1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-xckzd\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.317532 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.317516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/818162f8-f176-44b1-9fc0-2ac622116aa1-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-xckzd\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.324912 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.324865    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cntkr\" (UniqueName: \"kubernetes.io/projected/818162f8-f176-44b1-9fc0-2ac622116aa1-kube-api-access-cntkr\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-xckzd\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.358879 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.358864    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:02:56.646201 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.646174    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"]
Sep 16 18:02:56.648506 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:02:56.648479    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818162f8_f176_44b1_9fc0_2ac622116aa1.slice/crio-f1d89e52caa08fd65fda73f93a49a110d0a6eaf2170a66b4ce86c97da3f5d10a WatchSource:0}: Error finding container f1d89e52caa08fd65fda73f93a49a110d0a6eaf2170a66b4ce86c97da3f5d10a: Status 404 returned error can't find the container with id f1d89e52caa08fd65fda73f93a49a110d0a6eaf2170a66b4ce86c97da3f5d10a
Sep 16 18:02:56.745202 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.745175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" event={"ID":"818162f8-f176-44b1-9fc0-2ac622116aa1","Type":"ContainerStarted","Data":"ee81daa801dcd872b99b3f3c57298636015d18cf82933ccd621c8a0ff029d416"}
Sep 16 18:02:56.745553 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.745233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" event={"ID":"818162f8-f176-44b1-9fc0-2ac622116aa1","Type":"ContainerStarted","Data":"f1d89e52caa08fd65fda73f93a49a110d0a6eaf2170a66b4ce86c97da3f5d10a"}
Sep 16 18:02:56.747615 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.747593    2581 generic.go:356] "Generic (PLEG): container finished" podID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerID="1ca9ac0d799396fcf793234bf811a0526a7d0fe1cab0a4e76e7e078df653cb69" exitCode=2
Sep 16 18:02:56.747702 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:56.747638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" event={"ID":"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff","Type":"ContainerDied","Data":"1ca9ac0d799396fcf793234bf811a0526a7d0fe1cab0a4e76e7e078df653cb69"}
Sep 16 18:02:57.219203 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:57.219154    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 16 18:02:57.223634 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:57.223608    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 18:02:59.548950 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.548929    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:02:59.641741 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.641680    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv7v9\" (UniqueName: \"kubernetes.io/projected/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kube-api-access-mv7v9\") pod \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") "
Sep 16 18:02:59.641876 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.641748    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-cabundle-cert\") pod \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") "
Sep 16 18:02:59.641876 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.641769    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kube-rbac-proxy-sar-config\") pod \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") "
Sep 16 18:02:59.641876 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.641788    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-proxy-tls\") pod \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") "
Sep 16 18:02:59.641876 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.641816    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kserve-provision-location\") pod \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\" (UID: \"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff\") "
Sep 16 18:02:59.642195 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.642170    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" (UID: "732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:02:59.642264 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.642203    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" (UID: "732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:02:59.642264 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.642205    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" (UID: "732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:02:59.643875 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.643853    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kube-api-access-mv7v9" (OuterVolumeSpecName: "kube-api-access-mv7v9") pod "732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" (UID: "732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff"). InnerVolumeSpecName "kube-api-access-mv7v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:02:59.643958 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.643857    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" (UID: "732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:02:59.743312 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.743287    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:02:59.743312 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.743309    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:02:59.743443 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.743322    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:02:59.743443 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.743336    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:02:59.743443 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.743346    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv7v9\" (UniqueName: \"kubernetes.io/projected/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff-kube-api-access-mv7v9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:02:59.760465 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.760440    2581 generic.go:356] "Generic (PLEG): container finished" podID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerID="3dd0cae3845e49098cf7d27a75705470117063622cc3a28793a8d0a5eb079689" exitCode=0
Sep 16 18:02:59.760560 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.760497    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"
Sep 16 18:02:59.760560 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.760526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" event={"ID":"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff","Type":"ContainerDied","Data":"3dd0cae3845e49098cf7d27a75705470117063622cc3a28793a8d0a5eb079689"}
Sep 16 18:02:59.760675 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.760660    2581 scope.go:117] "RemoveContainer" containerID="1ca9ac0d799396fcf793234bf811a0526a7d0fe1cab0a4e76e7e078df653cb69"
Sep 16 18:02:59.766300 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.766272    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds" event={"ID":"732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff","Type":"ContainerDied","Data":"5627eec470ad25a0af9ec4862e906f562f403f39d2e4dc6b69fbeb56166c7aee"}
Sep 16 18:02:59.775907 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.775868    2581 scope.go:117] "RemoveContainer" containerID="3dd0cae3845e49098cf7d27a75705470117063622cc3a28793a8d0a5eb079689"
Sep 16 18:02:59.783682 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.783665    2581 scope.go:117] "RemoveContainer" containerID="0c16011fa6ae1c316f7bd0528622ece029609fad9b9f5a2c100e995002aa5fce"
Sep 16 18:02:59.790603 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.790586    2581 scope.go:117] "RemoveContainer" containerID="1ca9ac0d799396fcf793234bf811a0526a7d0fe1cab0a4e76e7e078df653cb69"
Sep 16 18:02:59.790836 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:02:59.790819    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca9ac0d799396fcf793234bf811a0526a7d0fe1cab0a4e76e7e078df653cb69\": container with ID starting with 1ca9ac0d799396fcf793234bf811a0526a7d0fe1cab0a4e76e7e078df653cb69 not found: ID does not exist" containerID="1ca9ac0d799396fcf793234bf811a0526a7d0fe1cab0a4e76e7e078df653cb69"
Sep 16 18:02:59.790925 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.790844    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca9ac0d799396fcf793234bf811a0526a7d0fe1cab0a4e76e7e078df653cb69"} err="failed to get container status \"1ca9ac0d799396fcf793234bf811a0526a7d0fe1cab0a4e76e7e078df653cb69\": rpc error: code = NotFound desc = could not find container \"1ca9ac0d799396fcf793234bf811a0526a7d0fe1cab0a4e76e7e078df653cb69\": container with ID starting with 1ca9ac0d799396fcf793234bf811a0526a7d0fe1cab0a4e76e7e078df653cb69 not found: ID does not exist"
Sep 16 18:02:59.790925 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.790859    2581 scope.go:117] "RemoveContainer" containerID="3dd0cae3845e49098cf7d27a75705470117063622cc3a28793a8d0a5eb079689"
Sep 16 18:02:59.791107 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:02:59.791092    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd0cae3845e49098cf7d27a75705470117063622cc3a28793a8d0a5eb079689\": container with ID starting with 3dd0cae3845e49098cf7d27a75705470117063622cc3a28793a8d0a5eb079689 not found: ID does not exist" containerID="3dd0cae3845e49098cf7d27a75705470117063622cc3a28793a8d0a5eb079689"
Sep 16 18:02:59.791149 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.791111    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd0cae3845e49098cf7d27a75705470117063622cc3a28793a8d0a5eb079689"} err="failed to get container status \"3dd0cae3845e49098cf7d27a75705470117063622cc3a28793a8d0a5eb079689\": rpc error: code = NotFound desc = could not find container \"3dd0cae3845e49098cf7d27a75705470117063622cc3a28793a8d0a5eb079689\": container with ID starting with 3dd0cae3845e49098cf7d27a75705470117063622cc3a28793a8d0a5eb079689 not found: ID does not exist"
Sep 16 18:02:59.791149 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.791123    2581 scope.go:117] "RemoveContainer" containerID="0c16011fa6ae1c316f7bd0528622ece029609fad9b9f5a2c100e995002aa5fce"
Sep 16 18:02:59.791367 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:02:59.791349    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c16011fa6ae1c316f7bd0528622ece029609fad9b9f5a2c100e995002aa5fce\": container with ID starting with 0c16011fa6ae1c316f7bd0528622ece029609fad9b9f5a2c100e995002aa5fce not found: ID does not exist" containerID="0c16011fa6ae1c316f7bd0528622ece029609fad9b9f5a2c100e995002aa5fce"
Sep 16 18:02:59.791416 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.791371    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c16011fa6ae1c316f7bd0528622ece029609fad9b9f5a2c100e995002aa5fce"} err="failed to get container status \"0c16011fa6ae1c316f7bd0528622ece029609fad9b9f5a2c100e995002aa5fce\": rpc error: code = NotFound desc = could not find container \"0c16011fa6ae1c316f7bd0528622ece029609fad9b9f5a2c100e995002aa5fce\": container with ID starting with 0c16011fa6ae1c316f7bd0528622ece029609fad9b9f5a2c100e995002aa5fce not found: ID does not exist"
Sep 16 18:02:59.799595 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.799575    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"]
Sep 16 18:02:59.808806 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:02:59.808782    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-cdzds"]
Sep 16 18:03:00.766423 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:00.766395    2581 generic.go:356] "Generic (PLEG): container finished" podID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerID="ee81daa801dcd872b99b3f3c57298636015d18cf82933ccd621c8a0ff029d416" exitCode=0
Sep 16 18:03:00.766745 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:00.766464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" event={"ID":"818162f8-f176-44b1-9fc0-2ac622116aa1","Type":"ContainerDied","Data":"ee81daa801dcd872b99b3f3c57298636015d18cf82933ccd621c8a0ff029d416"}
Sep 16 18:03:01.767616 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:01.767588    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" path="/var/lib/kubelet/pods/732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff/volumes"
Sep 16 18:03:01.772953 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:01.772923    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" event={"ID":"818162f8-f176-44b1-9fc0-2ac622116aa1","Type":"ContainerStarted","Data":"d1c1dbd58121bf271cea38842576831dcc4abf88a2c59768356fc0f6311dab23"}
Sep 16 18:03:01.773077 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:01.772979    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" event={"ID":"818162f8-f176-44b1-9fc0-2ac622116aa1","Type":"ContainerStarted","Data":"a61fa8d6a5fc1c721eb482dd91d5ffa81f1ab7aedd332813e7cbed827d1fca0b"}
Sep 16 18:03:01.773247 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:01.773231    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:03:01.773383 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:01.773365    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:03:01.774311 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:01.774288    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 16 18:03:02.777200 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:02.777167    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 16 18:03:07.782181 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:07.782156    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:03:07.782645 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:07.782570    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 16 18:03:07.820786 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:07.820742    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" podStartSLOduration=8.918140172 podStartE2EDuration="12.820728176s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.902588004s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:03:01.820925727 +0000 UTC m=+2310.609548257" watchObservedRunningTime="2026-09-16 18:03:07.820728176 +0000 UTC m=+2316.609350709"
Sep 16 18:03:17.782961 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:17.782922    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 16 18:03:24.993185 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:24.993113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:27.783228 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:27.783192    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 16 18:03:28.377698 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:28.377663    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:33.418632 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.418599    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:33.471935 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.471907    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"]
Sep 16 18:03:33.472264 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.472251    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kserve-container"
Sep 16 18:03:33.472307 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.472268    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kserve-container"
Sep 16 18:03:33.472307 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.472279    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="storage-initializer"
Sep 16 18:03:33.472307 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.472284    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="storage-initializer"
Sep 16 18:03:33.472307 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.472296    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kube-rbac-proxy"
Sep 16 18:03:33.472307 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.472303    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kube-rbac-proxy"
Sep 16 18:03:33.472462 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.472361    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kserve-container"
Sep 16 18:03:33.472462 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.472374    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="732c0b9d-bd3b-4cbc-8f16-b2fa0dbf09ff" containerName="kube-rbac-proxy"
Sep 16 18:03:33.475690 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.475673    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.489752 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.489734    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 16 18:03:33.489864 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.489758    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 18:03:33.504737 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.504717    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"]
Sep 16 18:03:33.579027 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.579005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dc2d520-f154-49c3-83fe-6d1fe738197d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.579146 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.579035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dc2d520-f154-49c3-83fe-6d1fe738197d-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.579146 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.579060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dc2d520-f154-49c3-83fe-6d1fe738197d-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.579146 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.579114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dc2d520-f154-49c3-83fe-6d1fe738197d-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.579307 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.579167    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhc8k\" (UniqueName: \"kubernetes.io/projected/5dc2d520-f154-49c3-83fe-6d1fe738197d-kube-api-access-dhc8k\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.611042 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.611016    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:33.680355 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.680298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dc2d520-f154-49c3-83fe-6d1fe738197d-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.680455 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.680374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhc8k\" (UniqueName: \"kubernetes.io/projected/5dc2d520-f154-49c3-83fe-6d1fe738197d-kube-api-access-dhc8k\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.680455 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.680434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dc2d520-f154-49c3-83fe-6d1fe738197d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.680579 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.680467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dc2d520-f154-49c3-83fe-6d1fe738197d-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.680579 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.680499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dc2d520-f154-49c3-83fe-6d1fe738197d-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.680704 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.680684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dc2d520-f154-49c3-83fe-6d1fe738197d-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.681154 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.681134    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dc2d520-f154-49c3-83fe-6d1fe738197d-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.681244 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.681141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dc2d520-f154-49c3-83fe-6d1fe738197d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.682972 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.682947    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dc2d520-f154-49c3-83fe-6d1fe738197d-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.691828 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.691811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhc8k\" (UniqueName: \"kubernetes.io/projected/5dc2d520-f154-49c3-83fe-6d1fe738197d-kube-api-access-dhc8k\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:33.784675 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:33.784657    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:34.060633 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:34.060604    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"]
Sep 16 18:03:34.062257 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:03:34.062232    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc2d520_f154_49c3_83fe_6d1fe738197d.slice/crio-aeddefa88597a0cccf33a0eb55fd5f0f80777c182c39bad542df9eefd88fd0a0 WatchSource:0}: Error finding container aeddefa88597a0cccf33a0eb55fd5f0f80777c182c39bad542df9eefd88fd0a0: Status 404 returned error can't find the container with id aeddefa88597a0cccf33a0eb55fd5f0f80777c182c39bad542df9eefd88fd0a0
Sep 16 18:03:34.899691 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:34.899657    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb" event={"ID":"5dc2d520-f154-49c3-83fe-6d1fe738197d","Type":"ContainerStarted","Data":"f95cf520cfc258dd46e443f1aa78b830b63bfb0838274e4041a07b44b82d73de"}
Sep 16 18:03:34.900083 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:34.899712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb" event={"ID":"5dc2d520-f154-49c3-83fe-6d1fe738197d","Type":"ContainerStarted","Data":"aeddefa88597a0cccf33a0eb55fd5f0f80777c182c39bad542df9eefd88fd0a0"}
Sep 16 18:03:37.782727 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:37.782692    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 16 18:03:37.913096 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:37.913069    2581 generic.go:356] "Generic (PLEG): container finished" podID="5dc2d520-f154-49c3-83fe-6d1fe738197d" containerID="f95cf520cfc258dd46e443f1aa78b830b63bfb0838274e4041a07b44b82d73de" exitCode=0
Sep 16 18:03:37.913197 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:37.913103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb" event={"ID":"5dc2d520-f154-49c3-83fe-6d1fe738197d","Type":"ContainerDied","Data":"f95cf520cfc258dd46e443f1aa78b830b63bfb0838274e4041a07b44b82d73de"}
Sep 16 18:03:38.920015 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:38.919977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb" event={"ID":"5dc2d520-f154-49c3-83fe-6d1fe738197d","Type":"ContainerStarted","Data":"20a0496c5811476bdd8f82322e1967d7c6feb9f2f88b5ec9ca4fb4cf0c920814"}
Sep 16 18:03:38.920417 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:38.920064    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb" event={"ID":"5dc2d520-f154-49c3-83fe-6d1fe738197d","Type":"ContainerStarted","Data":"686df4c1484a3f1e241c10be61ecdb5c387fb7a60aa759d61cc372ce1c7567a7"}
Sep 16 18:03:38.920417 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:38.920255    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:38.944582 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:38.944539    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb" podStartSLOduration=2.174958565 podStartE2EDuration="5.944525646s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.769567081s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:03:38.943199481 +0000 UTC m=+2347.731822013" watchObservedRunningTime="2026-09-16 18:03:38.944525646 +0000 UTC m=+2347.733148178"
Sep 16 18:03:39.925282 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:39.925247    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:45.935992 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:45.935959    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:03:47.782666 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:47.782623    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 16 18:03:53.654407 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:53.654373    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:57.783465 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:03:57.783422    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 16 18:04:07.783690 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:07.783661    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:04:15.812782 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:15.812752    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"]
Sep 16 18:04:15.813219 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:15.812988    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kserve-container" containerID="cri-o://a61fa8d6a5fc1c721eb482dd91d5ffa81f1ab7aedd332813e7cbed827d1fca0b" gracePeriod=30
Sep 16 18:04:15.813219 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:15.813027    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kube-rbac-proxy" containerID="cri-o://d1c1dbd58121bf271cea38842576831dcc4abf88a2c59768356fc0f6311dab23" gracePeriod=30
Sep 16 18:04:15.838743 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:15.838719    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:15.940192 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:15.940167    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:04:16.071615 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:16.071541    2581 generic.go:356] "Generic (PLEG): container finished" podID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerID="d1c1dbd58121bf271cea38842576831dcc4abf88a2c59768356fc0f6311dab23" exitCode=2
Sep 16 18:04:16.071615 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:16.071596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" event={"ID":"818162f8-f176-44b1-9fc0-2ac622116aa1","Type":"ContainerDied","Data":"d1c1dbd58121bf271cea38842576831dcc4abf88a2c59768356fc0f6311dab23"}
Sep 16 18:04:17.777595 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:17.777560    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Sep 16 18:04:17.782987 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:17.782965    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 16 18:04:19.455984 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.455960    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:04:19.612834 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.612809    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/818162f8-f176-44b1-9fc0-2ac622116aa1-kube-rbac-proxy-sar-config\") pod \"818162f8-f176-44b1-9fc0-2ac622116aa1\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") "
Sep 16 18:04:19.612981 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.612844    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/818162f8-f176-44b1-9fc0-2ac622116aa1-proxy-tls\") pod \"818162f8-f176-44b1-9fc0-2ac622116aa1\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") "
Sep 16 18:04:19.612981 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.612860    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/818162f8-f176-44b1-9fc0-2ac622116aa1-cabundle-cert\") pod \"818162f8-f176-44b1-9fc0-2ac622116aa1\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") "
Sep 16 18:04:19.612981 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.612883    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cntkr\" (UniqueName: \"kubernetes.io/projected/818162f8-f176-44b1-9fc0-2ac622116aa1-kube-api-access-cntkr\") pod \"818162f8-f176-44b1-9fc0-2ac622116aa1\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") "
Sep 16 18:04:19.612981 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.612925    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/818162f8-f176-44b1-9fc0-2ac622116aa1-kserve-provision-location\") pod \"818162f8-f176-44b1-9fc0-2ac622116aa1\" (UID: \"818162f8-f176-44b1-9fc0-2ac622116aa1\") "
Sep 16 18:04:19.613269 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.613243    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818162f8-f176-44b1-9fc0-2ac622116aa1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "818162f8-f176-44b1-9fc0-2ac622116aa1" (UID: "818162f8-f176-44b1-9fc0-2ac622116aa1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:04:19.613269 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.613252    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818162f8-f176-44b1-9fc0-2ac622116aa1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "818162f8-f176-44b1-9fc0-2ac622116aa1" (UID: "818162f8-f176-44b1-9fc0-2ac622116aa1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:04:19.613350 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.613289    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/818162f8-f176-44b1-9fc0-2ac622116aa1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "818162f8-f176-44b1-9fc0-2ac622116aa1" (UID: "818162f8-f176-44b1-9fc0-2ac622116aa1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:04:19.615112 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.615090    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818162f8-f176-44b1-9fc0-2ac622116aa1-kube-api-access-cntkr" (OuterVolumeSpecName: "kube-api-access-cntkr") pod "818162f8-f176-44b1-9fc0-2ac622116aa1" (UID: "818162f8-f176-44b1-9fc0-2ac622116aa1"). InnerVolumeSpecName "kube-api-access-cntkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:04:19.615168 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.615089    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/818162f8-f176-44b1-9fc0-2ac622116aa1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "818162f8-f176-44b1-9fc0-2ac622116aa1" (UID: "818162f8-f176-44b1-9fc0-2ac622116aa1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:04:19.713529 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.713503    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/818162f8-f176-44b1-9fc0-2ac622116aa1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:04:19.713529 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.713524    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/818162f8-f176-44b1-9fc0-2ac622116aa1-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:04:19.713660 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.713534    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/818162f8-f176-44b1-9fc0-2ac622116aa1-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:04:19.713660 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.713542    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cntkr\" (UniqueName: \"kubernetes.io/projected/818162f8-f176-44b1-9fc0-2ac622116aa1-kube-api-access-cntkr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:04:19.713660 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:19.713550    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/818162f8-f176-44b1-9fc0-2ac622116aa1-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:04:20.087437 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:20.087410    2581 generic.go:356] "Generic (PLEG): container finished" podID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerID="a61fa8d6a5fc1c721eb482dd91d5ffa81f1ab7aedd332813e7cbed827d1fca0b" exitCode=0
Sep 16 18:04:20.087554 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:20.087486    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"
Sep 16 18:04:20.087554 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:20.087493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" event={"ID":"818162f8-f176-44b1-9fc0-2ac622116aa1","Type":"ContainerDied","Data":"a61fa8d6a5fc1c721eb482dd91d5ffa81f1ab7aedd332813e7cbed827d1fca0b"}
Sep 16 18:04:20.087554 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:20.087544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd" event={"ID":"818162f8-f176-44b1-9fc0-2ac622116aa1","Type":"ContainerDied","Data":"f1d89e52caa08fd65fda73f93a49a110d0a6eaf2170a66b4ce86c97da3f5d10a"}
Sep 16 18:04:20.087719 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:20.087568    2581 scope.go:117] "RemoveContainer" containerID="d1c1dbd58121bf271cea38842576831dcc4abf88a2c59768356fc0f6311dab23"
Sep 16 18:04:20.095257 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:20.095240    2581 scope.go:117] "RemoveContainer" containerID="a61fa8d6a5fc1c721eb482dd91d5ffa81f1ab7aedd332813e7cbed827d1fca0b"
Sep 16 18:04:20.102267 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:20.102251    2581 scope.go:117] "RemoveContainer" containerID="ee81daa801dcd872b99b3f3c57298636015d18cf82933ccd621c8a0ff029d416"
Sep 16 18:04:20.109086 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:20.109054    2581 scope.go:117] "RemoveContainer" containerID="d1c1dbd58121bf271cea38842576831dcc4abf88a2c59768356fc0f6311dab23"
Sep 16 18:04:20.109492 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:04:20.109469    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c1dbd58121bf271cea38842576831dcc4abf88a2c59768356fc0f6311dab23\": container with ID starting with d1c1dbd58121bf271cea38842576831dcc4abf88a2c59768356fc0f6311dab23 not found: ID does not exist" containerID="d1c1dbd58121bf271cea38842576831dcc4abf88a2c59768356fc0f6311dab23"
Sep 16 18:04:20.109605 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:20.109502    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c1dbd58121bf271cea38842576831dcc4abf88a2c59768356fc0f6311dab23"} err="failed to get container status \"d1c1dbd58121bf271cea38842576831dcc4abf88a2c59768356fc0f6311dab23\": rpc error: code = NotFound desc = could not find container \"d1c1dbd58121bf271cea38842576831dcc4abf88a2c59768356fc0f6311dab23\": container with ID starting with d1c1dbd58121bf271cea38842576831dcc4abf88a2c59768356fc0f6311dab23 not found: ID does not exist"
Sep 16 18:04:20.109605 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:20.109524    2581 scope.go:117] "RemoveContainer" containerID="a61fa8d6a5fc1c721eb482dd91d5ffa81f1ab7aedd332813e7cbed827d1fca0b"
Sep 16 18:04:20.109812 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:04:20.109795    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61fa8d6a5fc1c721eb482dd91d5ffa81f1ab7aedd332813e7cbed827d1fca0b\": container with ID starting with a61fa8d6a5fc1c721eb482dd91d5ffa81f1ab7aedd332813e7cbed827d1fca0b not found: ID does not exist" containerID="a61fa8d6a5fc1c721eb482dd91d5ffa81f1ab7aedd332813e7cbed827d1fca0b"
Sep 16 18:04:20.109858 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:20.109818    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61fa8d6a5fc1c721eb482dd91d5ffa81f1ab7aedd332813e7cbed827d1fca0b"} err="failed to get container status \"a61fa8d6a5fc1c721eb482dd91d5ffa81f1ab7aedd332813e7cbed827d1fca0b\": rpc error: code = NotFound desc = could not find container \"a61fa8d6a5fc1c721eb482dd91d5ffa81f1ab7aedd332813e7cbed827d1fca0b\": container with ID starting with a61fa8d6a5fc1c721eb482dd91d5ffa81f1ab7aedd332813e7cbed827d1fca0b not found: ID does not exist"
Sep 16 18:04:20.109858 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:20.109835    2581 scope.go:117] "RemoveContainer" containerID="ee81daa801dcd872b99b3f3c57298636015d18cf82933ccd621c8a0ff029d416"
Sep 16 18:04:20.110162 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:04:20.110143    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee81daa801dcd872b99b3f3c57298636015d18cf82933ccd621c8a0ff029d416\": container with ID starting with ee81daa801dcd872b99b3f3c57298636015d18cf82933ccd621c8a0ff029d416 not found: ID does not exist" containerID="ee81daa801dcd872b99b3f3c57298636015d18cf82933ccd621c8a0ff029d416"
Sep 16 18:04:20.110243 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:20.110164    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee81daa801dcd872b99b3f3c57298636015d18cf82933ccd621c8a0ff029d416"} err="failed to get container status \"ee81daa801dcd872b99b3f3c57298636015d18cf82933ccd621c8a0ff029d416\": rpc error: code = NotFound desc = could not find container \"ee81daa801dcd872b99b3f3c57298636015d18cf82933ccd621c8a0ff029d416\": container with ID starting with ee81daa801dcd872b99b3f3c57298636015d18cf82933ccd621c8a0ff029d416 not found: ID does not exist"
Sep 16 18:04:20.111342 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:20.111324    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"]
Sep 16 18:04:20.115578 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:20.115559    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-xckzd"]
Sep 16 18:04:21.042745 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.042714    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:21.162352 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.162328    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:21.292828 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.292756    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"]
Sep 16 18:04:21.293176 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.293159    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="storage-initializer"
Sep 16 18:04:21.293274 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.293180    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="storage-initializer"
Sep 16 18:04:21.293274 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.293190    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kube-rbac-proxy"
Sep 16 18:04:21.293274 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.293198    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kube-rbac-proxy"
Sep 16 18:04:21.293274 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.293220    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kserve-container"
Sep 16 18:04:21.293274 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.293229    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kserve-container"
Sep 16 18:04:21.293526 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.293322    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kube-rbac-proxy"
Sep 16 18:04:21.293526 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.293335    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" containerName="kserve-container"
Sep 16 18:04:21.297869 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.297849    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.301612 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.301594    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 18:04:21.302827 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.302809    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 18:04:21.314115 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.314095    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"]
Sep 16 18:04:21.423392 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.423368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd2b579d-d721-4ede-a501-99f75da66cde-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-b5wmv\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.423489 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.423397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd2b579d-d721-4ede-a501-99f75da66cde-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-b5wmv\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.423489 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.423431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd2b579d-d721-4ede-a501-99f75da66cde-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-b5wmv\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.423560 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.423493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd2b579d-d721-4ede-a501-99f75da66cde-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-b5wmv\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.423560 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.423533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdcz6\" (UniqueName: \"kubernetes.io/projected/bd2b579d-d721-4ede-a501-99f75da66cde-kube-api-access-fdcz6\") pod \"isvc-canary-stop-predictor-59fbfd5bc-b5wmv\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.524748 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.524722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd2b579d-d721-4ede-a501-99f75da66cde-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-b5wmv\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.524825 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.524754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd2b579d-d721-4ede-a501-99f75da66cde-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-b5wmv\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.524825 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.524779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdcz6\" (UniqueName: \"kubernetes.io/projected/bd2b579d-d721-4ede-a501-99f75da66cde-kube-api-access-fdcz6\") pod \"isvc-canary-stop-predictor-59fbfd5bc-b5wmv\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.524917 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.524826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd2b579d-d721-4ede-a501-99f75da66cde-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-b5wmv\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.524917 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.524850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd2b579d-d721-4ede-a501-99f75da66cde-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-b5wmv\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.525175 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.525149    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd2b579d-d721-4ede-a501-99f75da66cde-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-b5wmv\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.525544 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.525525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd2b579d-d721-4ede-a501-99f75da66cde-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-b5wmv\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.525544 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.525535    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd2b579d-d721-4ede-a501-99f75da66cde-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-b5wmv\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.527251 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.527235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd2b579d-d721-4ede-a501-99f75da66cde-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-b5wmv\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.537104 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.537086    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdcz6\" (UniqueName: \"kubernetes.io/projected/bd2b579d-d721-4ede-a501-99f75da66cde-kube-api-access-fdcz6\") pod \"isvc-canary-stop-predictor-59fbfd5bc-b5wmv\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.556006 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.555962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:21.608772 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.608748    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:21.622820 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.622799    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:21.766650 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.766623    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="818162f8-f176-44b1-9fc0-2ac622116aa1" path="/var/lib/kubelet/pods/818162f8-f176-44b1-9fc0-2ac622116aa1/volumes"
Sep 16 18:04:21.890426 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:21.890406    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"]
Sep 16 18:04:21.892685 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:04:21.892657    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2b579d_d721_4ede_a501_99f75da66cde.slice/crio-eddadad5f76b3a435dc42f1e17d3f833263f387689df57b023877584f114df3d WatchSource:0}: Error finding container eddadad5f76b3a435dc42f1e17d3f833263f387689df57b023877584f114df3d: Status 404 returned error can't find the container with id eddadad5f76b3a435dc42f1e17d3f833263f387689df57b023877584f114df3d
Sep 16 18:04:22.099552 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:22.099515    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" event={"ID":"bd2b579d-d721-4ede-a501-99f75da66cde","Type":"ContainerStarted","Data":"3a600556e978b6b65aa671f86e84c7d0a6aae939203ea1b7f9f3818dcecf530a"}
Sep 16 18:04:22.099911 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:22.099581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" event={"ID":"bd2b579d-d721-4ede-a501-99f75da66cde","Type":"ContainerStarted","Data":"eddadad5f76b3a435dc42f1e17d3f833263f387689df57b023877584f114df3d"}
Sep 16 18:04:23.528221 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:23.528191    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"]
Sep 16 18:04:23.528607 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:23.528409    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb" podUID="5dc2d520-f154-49c3-83fe-6d1fe738197d" containerName="kserve-container" containerID="cri-o://686df4c1484a3f1e241c10be61ecdb5c387fb7a60aa759d61cc372ce1c7567a7" gracePeriod=30
Sep 16 18:04:23.528607 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:23.528475    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb" podUID="5dc2d520-f154-49c3-83fe-6d1fe738197d" containerName="kube-rbac-proxy" containerID="cri-o://20a0496c5811476bdd8f82322e1967d7c6feb9f2f88b5ec9ca4fb4cf0c920814" gracePeriod=30
Sep 16 18:04:23.537278 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:23.537253    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:24.110704 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:24.110665    2581 generic.go:356] "Generic (PLEG): container finished" podID="5dc2d520-f154-49c3-83fe-6d1fe738197d" containerID="20a0496c5811476bdd8f82322e1967d7c6feb9f2f88b5ec9ca4fb4cf0c920814" exitCode=2
Sep 16 18:04:24.110864 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:24.110735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb" event={"ID":"5dc2d520-f154-49c3-83fe-6d1fe738197d","Type":"ContainerDied","Data":"20a0496c5811476bdd8f82322e1967d7c6feb9f2f88b5ec9ca4fb4cf0c920814"}
Sep 16 18:04:25.932251 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:25.932217    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb" podUID="5dc2d520-f154-49c3-83fe-6d1fe738197d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 16 18:04:25.936605 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:25.936577    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb" podUID="5dc2d520-f154-49c3-83fe-6d1fe738197d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 16 18:04:26.120917 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:26.120875    2581 generic.go:356] "Generic (PLEG): container finished" podID="bd2b579d-d721-4ede-a501-99f75da66cde" containerID="3a600556e978b6b65aa671f86e84c7d0a6aae939203ea1b7f9f3818dcecf530a" exitCode=0
Sep 16 18:04:26.121028 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:26.120939    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" event={"ID":"bd2b579d-d721-4ede-a501-99f75da66cde","Type":"ContainerDied","Data":"3a600556e978b6b65aa671f86e84c7d0a6aae939203ea1b7f9f3818dcecf530a"}
Sep 16 18:04:27.128458 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:27.128426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" event={"ID":"bd2b579d-d721-4ede-a501-99f75da66cde","Type":"ContainerStarted","Data":"60188014cd0cacb276acce39260643db578097933537de12833d92dc4744e508"}
Sep 16 18:04:27.128841 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:27.128489    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" event={"ID":"bd2b579d-d721-4ede-a501-99f75da66cde","Type":"ContainerStarted","Data":"7080ec9de0638dc331344dc5a906bcbba177aa801e1462811f01c6f7dbef5742"}
Sep 16 18:04:27.128841 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:27.128743    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:27.128841 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:27.128798    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:27.130102 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:27.130076    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 18:04:27.154457 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:27.154410    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" podStartSLOduration=2.211523276 podStartE2EDuration="6.1543969s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.942873624s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:04:27.152111614 +0000 UTC m=+2395.940734146" watchObservedRunningTime="2026-09-16 18:04:27.1543969 +0000 UTC m=+2395.943019433"
Sep 16 18:04:28.132759 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.132718    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 18:04:28.373117 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.373094    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:04:28.478783 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.478726    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dc2d520-f154-49c3-83fe-6d1fe738197d-kube-rbac-proxy-sar-config\") pod \"5dc2d520-f154-49c3-83fe-6d1fe738197d\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") "
Sep 16 18:04:28.478950 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.478791    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dc2d520-f154-49c3-83fe-6d1fe738197d-proxy-tls\") pod \"5dc2d520-f154-49c3-83fe-6d1fe738197d\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") "
Sep 16 18:04:28.478950 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.478841    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dc2d520-f154-49c3-83fe-6d1fe738197d-cabundle-cert\") pod \"5dc2d520-f154-49c3-83fe-6d1fe738197d\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") "
Sep 16 18:04:28.478950 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.478858    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhc8k\" (UniqueName: \"kubernetes.io/projected/5dc2d520-f154-49c3-83fe-6d1fe738197d-kube-api-access-dhc8k\") pod \"5dc2d520-f154-49c3-83fe-6d1fe738197d\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") "
Sep 16 18:04:28.478950 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.478873    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dc2d520-f154-49c3-83fe-6d1fe738197d-kserve-provision-location\") pod \"5dc2d520-f154-49c3-83fe-6d1fe738197d\" (UID: \"5dc2d520-f154-49c3-83fe-6d1fe738197d\") "
Sep 16 18:04:28.479245 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.479222    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc2d520-f154-49c3-83fe-6d1fe738197d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5dc2d520-f154-49c3-83fe-6d1fe738197d" (UID: "5dc2d520-f154-49c3-83fe-6d1fe738197d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:04:28.479301 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.479230    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc2d520-f154-49c3-83fe-6d1fe738197d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5dc2d520-f154-49c3-83fe-6d1fe738197d" (UID: "5dc2d520-f154-49c3-83fe-6d1fe738197d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:04:28.479301 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.479237    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc2d520-f154-49c3-83fe-6d1fe738197d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5dc2d520-f154-49c3-83fe-6d1fe738197d" (UID: "5dc2d520-f154-49c3-83fe-6d1fe738197d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:04:28.480967 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.480948    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dc2d520-f154-49c3-83fe-6d1fe738197d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5dc2d520-f154-49c3-83fe-6d1fe738197d" (UID: "5dc2d520-f154-49c3-83fe-6d1fe738197d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:04:28.481042 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.481018    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc2d520-f154-49c3-83fe-6d1fe738197d-kube-api-access-dhc8k" (OuterVolumeSpecName: "kube-api-access-dhc8k") pod "5dc2d520-f154-49c3-83fe-6d1fe738197d" (UID: "5dc2d520-f154-49c3-83fe-6d1fe738197d"). InnerVolumeSpecName "kube-api-access-dhc8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:04:28.580182 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.580163    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dc2d520-f154-49c3-83fe-6d1fe738197d-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:04:28.580182 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.580182    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhc8k\" (UniqueName: \"kubernetes.io/projected/5dc2d520-f154-49c3-83fe-6d1fe738197d-kube-api-access-dhc8k\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:04:28.580304 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.580191    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dc2d520-f154-49c3-83fe-6d1fe738197d-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:04:28.580304 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.580199    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dc2d520-f154-49c3-83fe-6d1fe738197d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:04:28.580304 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.580208    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dc2d520-f154-49c3-83fe-6d1fe738197d-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:04:28.700754 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.700723    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:28.756137 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.756077    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"]
Sep 16 18:04:28.756529 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.756516    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc2d520-f154-49c3-83fe-6d1fe738197d" containerName="kserve-container"
Sep 16 18:04:28.756571 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.756533    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc2d520-f154-49c3-83fe-6d1fe738197d" containerName="kserve-container"
Sep 16 18:04:28.756571 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.756555    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc2d520-f154-49c3-83fe-6d1fe738197d" containerName="kube-rbac-proxy"
Sep 16 18:04:28.756571 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.756563    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc2d520-f154-49c3-83fe-6d1fe738197d" containerName="kube-rbac-proxy"
Sep 16 18:04:28.756664 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.756585    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc2d520-f154-49c3-83fe-6d1fe738197d" containerName="storage-initializer"
Sep 16 18:04:28.756664 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.756595    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc2d520-f154-49c3-83fe-6d1fe738197d" containerName="storage-initializer"
Sep 16 18:04:28.756727 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.756667    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc2d520-f154-49c3-83fe-6d1fe738197d" containerName="kserve-container"
Sep 16 18:04:28.756727 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.756679    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc2d520-f154-49c3-83fe-6d1fe738197d" containerName="kube-rbac-proxy"
Sep 16 18:04:28.760391 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.760369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:28.762988 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.762971    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 16 18:04:28.764240 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.764223    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 18:04:28.772765 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.772745    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"]
Sep 16 18:04:28.876064 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.876032    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:28.882143 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.882118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/001af213-15f1-45d8-910b-63f67d62b2a3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:28.882223 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.882152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/001af213-15f1-45d8-910b-63f67d62b2a3-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:28.882276 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.882226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ktvr\" (UniqueName: \"kubernetes.io/projected/001af213-15f1-45d8-910b-63f67d62b2a3-kube-api-access-6ktvr\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:28.882276 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.882253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/001af213-15f1-45d8-910b-63f67d62b2a3-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:28.882364 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.882280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/001af213-15f1-45d8-910b-63f67d62b2a3-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:28.983128 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.983104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/001af213-15f1-45d8-910b-63f67d62b2a3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:28.983230 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.983134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/001af213-15f1-45d8-910b-63f67d62b2a3-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:28.983230 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.983196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ktvr\" (UniqueName: \"kubernetes.io/projected/001af213-15f1-45d8-910b-63f67d62b2a3-kube-api-access-6ktvr\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:28.983301 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.983230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/001af213-15f1-45d8-910b-63f67d62b2a3-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:28.983301 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.983258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/001af213-15f1-45d8-910b-63f67d62b2a3-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:28.983680 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.983658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/001af213-15f1-45d8-910b-63f67d62b2a3-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:28.983953 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.983909    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/001af213-15f1-45d8-910b-63f67d62b2a3-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:28.984014 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.983969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/001af213-15f1-45d8-910b-63f67d62b2a3-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:28.985424 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.985408    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/001af213-15f1-45d8-910b-63f67d62b2a3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:28.993358 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:28.993340    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ktvr\" (UniqueName: \"kubernetes.io/projected/001af213-15f1-45d8-910b-63f67d62b2a3-kube-api-access-6ktvr\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:29.073103 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.073081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:29.146347 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.146313    2581 generic.go:356] "Generic (PLEG): container finished" podID="5dc2d520-f154-49c3-83fe-6d1fe738197d" containerID="686df4c1484a3f1e241c10be61ecdb5c387fb7a60aa759d61cc372ce1c7567a7" exitCode=0
Sep 16 18:04:29.146676 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.146395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb" event={"ID":"5dc2d520-f154-49c3-83fe-6d1fe738197d","Type":"ContainerDied","Data":"686df4c1484a3f1e241c10be61ecdb5c387fb7a60aa759d61cc372ce1c7567a7"}
Sep 16 18:04:29.146676 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.146398    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"
Sep 16 18:04:29.146676 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.146478    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb" event={"ID":"5dc2d520-f154-49c3-83fe-6d1fe738197d","Type":"ContainerDied","Data":"aeddefa88597a0cccf33a0eb55fd5f0f80777c182c39bad542df9eefd88fd0a0"}
Sep 16 18:04:29.146676 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.146503    2581 scope.go:117] "RemoveContainer" containerID="20a0496c5811476bdd8f82322e1967d7c6feb9f2f88b5ec9ca4fb4cf0c920814"
Sep 16 18:04:29.155784 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.155758    2581 scope.go:117] "RemoveContainer" containerID="686df4c1484a3f1e241c10be61ecdb5c387fb7a60aa759d61cc372ce1c7567a7"
Sep 16 18:04:29.163794 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.163764    2581 scope.go:117] "RemoveContainer" containerID="f95cf520cfc258dd46e443f1aa78b830b63bfb0838274e4041a07b44b82d73de"
Sep 16 18:04:29.171949 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.171929    2581 scope.go:117] "RemoveContainer" containerID="20a0496c5811476bdd8f82322e1967d7c6feb9f2f88b5ec9ca4fb4cf0c920814"
Sep 16 18:04:29.172341 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.172312    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"]
Sep 16 18:04:29.172814 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:04:29.172795    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a0496c5811476bdd8f82322e1967d7c6feb9f2f88b5ec9ca4fb4cf0c920814\": container with ID starting with 20a0496c5811476bdd8f82322e1967d7c6feb9f2f88b5ec9ca4fb4cf0c920814 not found: ID does not exist" containerID="20a0496c5811476bdd8f82322e1967d7c6feb9f2f88b5ec9ca4fb4cf0c920814"
Sep 16 18:04:29.172917 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.172822    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a0496c5811476bdd8f82322e1967d7c6feb9f2f88b5ec9ca4fb4cf0c920814"} err="failed to get container status \"20a0496c5811476bdd8f82322e1967d7c6feb9f2f88b5ec9ca4fb4cf0c920814\": rpc error: code = NotFound desc = could not find container \"20a0496c5811476bdd8f82322e1967d7c6feb9f2f88b5ec9ca4fb4cf0c920814\": container with ID starting with 20a0496c5811476bdd8f82322e1967d7c6feb9f2f88b5ec9ca4fb4cf0c920814 not found: ID does not exist"
Sep 16 18:04:29.172917 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.172837    2581 scope.go:117] "RemoveContainer" containerID="686df4c1484a3f1e241c10be61ecdb5c387fb7a60aa759d61cc372ce1c7567a7"
Sep 16 18:04:29.173180 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:04:29.173161    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686df4c1484a3f1e241c10be61ecdb5c387fb7a60aa759d61cc372ce1c7567a7\": container with ID starting with 686df4c1484a3f1e241c10be61ecdb5c387fb7a60aa759d61cc372ce1c7567a7 not found: ID does not exist" containerID="686df4c1484a3f1e241c10be61ecdb5c387fb7a60aa759d61cc372ce1c7567a7"
Sep 16 18:04:29.173238 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.173195    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686df4c1484a3f1e241c10be61ecdb5c387fb7a60aa759d61cc372ce1c7567a7"} err="failed to get container status \"686df4c1484a3f1e241c10be61ecdb5c387fb7a60aa759d61cc372ce1c7567a7\": rpc error: code = NotFound desc = could not find container \"686df4c1484a3f1e241c10be61ecdb5c387fb7a60aa759d61cc372ce1c7567a7\": container with ID starting with 686df4c1484a3f1e241c10be61ecdb5c387fb7a60aa759d61cc372ce1c7567a7 not found: ID does not exist"
Sep 16 18:04:29.173238 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.173215    2581 scope.go:117] "RemoveContainer" containerID="f95cf520cfc258dd46e443f1aa78b830b63bfb0838274e4041a07b44b82d73de"
Sep 16 18:04:29.173468 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:04:29.173446    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f95cf520cfc258dd46e443f1aa78b830b63bfb0838274e4041a07b44b82d73de\": container with ID starting with f95cf520cfc258dd46e443f1aa78b830b63bfb0838274e4041a07b44b82d73de not found: ID does not exist" containerID="f95cf520cfc258dd46e443f1aa78b830b63bfb0838274e4041a07b44b82d73de"
Sep 16 18:04:29.173507 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.173474    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f95cf520cfc258dd46e443f1aa78b830b63bfb0838274e4041a07b44b82d73de"} err="failed to get container status \"f95cf520cfc258dd46e443f1aa78b830b63bfb0838274e4041a07b44b82d73de\": rpc error: code = NotFound desc = could not find container \"f95cf520cfc258dd46e443f1aa78b830b63bfb0838274e4041a07b44b82d73de\": container with ID starting with f95cf520cfc258dd46e443f1aa78b830b63bfb0838274e4041a07b44b82d73de not found: ID does not exist"
Sep 16 18:04:29.174835 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.174814    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-b24xb"]
Sep 16 18:04:29.349362 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.349339    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"]
Sep 16 18:04:29.350560 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:04:29.350535    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001af213_15f1_45d8_910b_63f67d62b2a3.slice/crio-7141bebcacaae94d9f60bab38d58c147442ee5b2fdf09a579bb365ef3b7322e0 WatchSource:0}: Error finding container 7141bebcacaae94d9f60bab38d58c147442ee5b2fdf09a579bb365ef3b7322e0: Status 404 returned error can't find the container with id 7141bebcacaae94d9f60bab38d58c147442ee5b2fdf09a579bb365ef3b7322e0
Sep 16 18:04:29.768049 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:29.767978    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dc2d520-f154-49c3-83fe-6d1fe738197d" path="/var/lib/kubelet/pods/5dc2d520-f154-49c3-83fe-6d1fe738197d/volumes"
Sep 16 18:04:30.152189 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:30.152159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm" event={"ID":"001af213-15f1-45d8-910b-63f67d62b2a3","Type":"ContainerStarted","Data":"e734704ac0fc5a362bc5c93fd2ee02f5e840bcda97cd20321c9b8ec537d91b7c"}
Sep 16 18:04:30.152626 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:30.152225    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm" event={"ID":"001af213-15f1-45d8-910b-63f67d62b2a3","Type":"ContainerStarted","Data":"7141bebcacaae94d9f60bab38d58c147442ee5b2fdf09a579bb365ef3b7322e0"}
Sep 16 18:04:31.862769 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:31.862729    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 18:04:31.864711 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:31.864690    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 18:04:33.137222 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:33.137200    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:04:33.137732 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:33.137705    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 18:04:33.167862 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:33.167833    2581 generic.go:356] "Generic (PLEG): container finished" podID="001af213-15f1-45d8-910b-63f67d62b2a3" containerID="e734704ac0fc5a362bc5c93fd2ee02f5e840bcda97cd20321c9b8ec537d91b7c" exitCode=0
Sep 16 18:04:33.167978 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:33.167921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm" event={"ID":"001af213-15f1-45d8-910b-63f67d62b2a3","Type":"ContainerDied","Data":"e734704ac0fc5a362bc5c93fd2ee02f5e840bcda97cd20321c9b8ec537d91b7c"}
Sep 16 18:04:34.174318 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:34.174288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm" event={"ID":"001af213-15f1-45d8-910b-63f67d62b2a3","Type":"ContainerStarted","Data":"812b971dc18efc9edb045fd06fdeb7992fe6ef0d0483ecdd84b05c48940a691e"}
Sep 16 18:04:34.174653 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:34.174345    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm" event={"ID":"001af213-15f1-45d8-910b-63f67d62b2a3","Type":"ContainerStarted","Data":"373b450da6df74bb19894f8a4a5667ee096a6f2e60bbfe8db81e752f281c7b70"}
Sep 16 18:04:34.174653 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:34.174527    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:34.201922 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:34.201864    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm" podStartSLOduration=2.486682776 podStartE2EDuration="6.20185133s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.715168554s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:04:34.200294348 +0000 UTC m=+2402.988916879" watchObservedRunningTime="2026-09-16 18:04:34.20185133 +0000 UTC m=+2402.990473862"
Sep 16 18:04:35.179102 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:35.179073    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:41.189142 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:41.189109    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:04:43.138351 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:43.138309    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 18:04:53.138326 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:04:53.138249    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 18:05:03.138611 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:03.138576    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 18:05:11.192819 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:11.192786    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:05:13.137779 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:13.137744    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 18:05:18.864440 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:18.864403    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"]
Sep 16 18:05:18.864845 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:18.864691    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm" podUID="001af213-15f1-45d8-910b-63f67d62b2a3" containerName="kserve-container" containerID="cri-o://373b450da6df74bb19894f8a4a5667ee096a6f2e60bbfe8db81e752f281c7b70" gracePeriod=30
Sep 16 18:05:18.864845 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:18.864716    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm" podUID="001af213-15f1-45d8-910b-63f67d62b2a3" containerName="kube-rbac-proxy" containerID="cri-o://812b971dc18efc9edb045fd06fdeb7992fe6ef0d0483ecdd84b05c48940a691e" gracePeriod=30
Sep 16 18:05:18.885204 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:18.885172    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:05:19.361884 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:19.361848    2581 generic.go:356] "Generic (PLEG): container finished" podID="001af213-15f1-45d8-910b-63f67d62b2a3" containerID="812b971dc18efc9edb045fd06fdeb7992fe6ef0d0483ecdd84b05c48940a691e" exitCode=2
Sep 16 18:05:19.362044 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:19.361928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm" event={"ID":"001af213-15f1-45d8-910b-63f67d62b2a3","Type":"ContainerDied","Data":"812b971dc18efc9edb045fd06fdeb7992fe6ef0d0483ecdd84b05c48940a691e"}
Sep 16 18:05:21.184406 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:21.184366    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm" podUID="001af213-15f1-45d8-910b-63f67d62b2a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Sep 16 18:05:21.189822 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:21.189799    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm" podUID="001af213-15f1-45d8-910b-63f67d62b2a3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.55:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 16 18:05:23.138530 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:23.138486    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 18:05:23.906550 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:23.906528    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:05:23.975051 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:23.975023    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/001af213-15f1-45d8-910b-63f67d62b2a3-kube-rbac-proxy-sar-config\") pod \"001af213-15f1-45d8-910b-63f67d62b2a3\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") "
Sep 16 18:05:23.975177 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:23.975094    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/001af213-15f1-45d8-910b-63f67d62b2a3-cabundle-cert\") pod \"001af213-15f1-45d8-910b-63f67d62b2a3\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") "
Sep 16 18:05:23.975243 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:23.975198    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/001af213-15f1-45d8-910b-63f67d62b2a3-proxy-tls\") pod \"001af213-15f1-45d8-910b-63f67d62b2a3\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") "
Sep 16 18:05:23.975303 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:23.975254    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ktvr\" (UniqueName: \"kubernetes.io/projected/001af213-15f1-45d8-910b-63f67d62b2a3-kube-api-access-6ktvr\") pod \"001af213-15f1-45d8-910b-63f67d62b2a3\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") "
Sep 16 18:05:23.975303 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:23.975281    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/001af213-15f1-45d8-910b-63f67d62b2a3-kserve-provision-location\") pod \"001af213-15f1-45d8-910b-63f67d62b2a3\" (UID: \"001af213-15f1-45d8-910b-63f67d62b2a3\") "
Sep 16 18:05:23.975412 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:23.975362    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001af213-15f1-45d8-910b-63f67d62b2a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "001af213-15f1-45d8-910b-63f67d62b2a3" (UID: "001af213-15f1-45d8-910b-63f67d62b2a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:05:23.975412 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:23.975401    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001af213-15f1-45d8-910b-63f67d62b2a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "001af213-15f1-45d8-910b-63f67d62b2a3" (UID: "001af213-15f1-45d8-910b-63f67d62b2a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:05:23.975535 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:23.975519    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/001af213-15f1-45d8-910b-63f67d62b2a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:05:23.975590 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:23.975541    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/001af213-15f1-45d8-910b-63f67d62b2a3-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:05:23.975635 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:23.975619    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001af213-15f1-45d8-910b-63f67d62b2a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "001af213-15f1-45d8-910b-63f67d62b2a3" (UID: "001af213-15f1-45d8-910b-63f67d62b2a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:05:23.977378 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:23.977348    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001af213-15f1-45d8-910b-63f67d62b2a3-kube-api-access-6ktvr" (OuterVolumeSpecName: "kube-api-access-6ktvr") pod "001af213-15f1-45d8-910b-63f67d62b2a3" (UID: "001af213-15f1-45d8-910b-63f67d62b2a3"). InnerVolumeSpecName "kube-api-access-6ktvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:05:23.977378 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:23.977356    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/001af213-15f1-45d8-910b-63f67d62b2a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "001af213-15f1-45d8-910b-63f67d62b2a3" (UID: "001af213-15f1-45d8-910b-63f67d62b2a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:05:24.076258 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.076230    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/001af213-15f1-45d8-910b-63f67d62b2a3-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:05:24.076258 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.076256    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ktvr\" (UniqueName: \"kubernetes.io/projected/001af213-15f1-45d8-910b-63f67d62b2a3-kube-api-access-6ktvr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:05:24.076413 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.076265    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/001af213-15f1-45d8-910b-63f67d62b2a3-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:05:24.089653 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.089631    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:05:24.164903 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.164863    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"]
Sep 16 18:05:24.165362 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.165344    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="001af213-15f1-45d8-910b-63f67d62b2a3" containerName="kserve-container"
Sep 16 18:05:24.165412 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.165367    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="001af213-15f1-45d8-910b-63f67d62b2a3" containerName="kserve-container"
Sep 16 18:05:24.165412 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.165396    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="001af213-15f1-45d8-910b-63f67d62b2a3" containerName="storage-initializer"
Sep 16 18:05:24.165412 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.165405    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="001af213-15f1-45d8-910b-63f67d62b2a3" containerName="storage-initializer"
Sep 16 18:05:24.165505 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.165432    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="001af213-15f1-45d8-910b-63f67d62b2a3" containerName="kube-rbac-proxy"
Sep 16 18:05:24.165505 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.165441    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="001af213-15f1-45d8-910b-63f67d62b2a3" containerName="kube-rbac-proxy"
Sep 16 18:05:24.165566 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.165511    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="001af213-15f1-45d8-910b-63f67d62b2a3" containerName="kserve-container"
Sep 16 18:05:24.165566 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.165524    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="001af213-15f1-45d8-910b-63f67d62b2a3" containerName="kube-rbac-proxy"
Sep 16 18:05:24.169139 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.169125    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.198019 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.198001    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 16 18:05:24.198139 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.198008    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 16 18:05:24.277528 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.277506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8mbp\" (UniqueName: \"kubernetes.io/projected/f270934c-6e6d-416f-87f7-700d9b34b259-kube-api-access-m8mbp\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-prfwv\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.277622 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.277530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f270934c-6e6d-416f-87f7-700d9b34b259-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-prfwv\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.277622 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.277550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f270934c-6e6d-416f-87f7-700d9b34b259-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-prfwv\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.277622 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.277566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f270934c-6e6d-416f-87f7-700d9b34b259-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-prfwv\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.277726 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.277703    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f270934c-6e6d-416f-87f7-700d9b34b259-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-prfwv\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.282753 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.282735    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:05:24.315595 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.315572    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"]
Sep 16 18:05:24.378917 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.378857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f270934c-6e6d-416f-87f7-700d9b34b259-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-prfwv\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.378917 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.378885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mbp\" (UniqueName: \"kubernetes.io/projected/f270934c-6e6d-416f-87f7-700d9b34b259-kube-api-access-m8mbp\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-prfwv\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.378917 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.378914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f270934c-6e6d-416f-87f7-700d9b34b259-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-prfwv\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.379140 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.378931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f270934c-6e6d-416f-87f7-700d9b34b259-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-prfwv\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.379140 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.378948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f270934c-6e6d-416f-87f7-700d9b34b259-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-prfwv\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.379418 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.379394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f270934c-6e6d-416f-87f7-700d9b34b259-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-prfwv\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.379586 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.379569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f270934c-6e6d-416f-87f7-700d9b34b259-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-prfwv\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.379650 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.379602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f270934c-6e6d-416f-87f7-700d9b34b259-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-prfwv\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.381356 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.381340    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f270934c-6e6d-416f-87f7-700d9b34b259-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-prfwv\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.388556 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.388535    2581 generic.go:356] "Generic (PLEG): container finished" podID="001af213-15f1-45d8-910b-63f67d62b2a3" containerID="373b450da6df74bb19894f8a4a5667ee096a6f2e60bbfe8db81e752f281c7b70" exitCode=0
Sep 16 18:05:24.388646 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.388587    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"
Sep 16 18:05:24.388646 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.388601    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm" event={"ID":"001af213-15f1-45d8-910b-63f67d62b2a3","Type":"ContainerDied","Data":"373b450da6df74bb19894f8a4a5667ee096a6f2e60bbfe8db81e752f281c7b70"}
Sep 16 18:05:24.388646 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.388641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm" event={"ID":"001af213-15f1-45d8-910b-63f67d62b2a3","Type":"ContainerDied","Data":"7141bebcacaae94d9f60bab38d58c147442ee5b2fdf09a579bb365ef3b7322e0"}
Sep 16 18:05:24.388751 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.388666    2581 scope.go:117] "RemoveContainer" containerID="812b971dc18efc9edb045fd06fdeb7992fe6ef0d0483ecdd84b05c48940a691e"
Sep 16 18:05:24.397114 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.397098    2581 scope.go:117] "RemoveContainer" containerID="373b450da6df74bb19894f8a4a5667ee096a6f2e60bbfe8db81e752f281c7b70"
Sep 16 18:05:24.404675 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.404659    2581 scope.go:117] "RemoveContainer" containerID="e734704ac0fc5a362bc5c93fd2ee02f5e840bcda97cd20321c9b8ec537d91b7c"
Sep 16 18:05:24.411490 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.411474    2581 scope.go:117] "RemoveContainer" containerID="812b971dc18efc9edb045fd06fdeb7992fe6ef0d0483ecdd84b05c48940a691e"
Sep 16 18:05:24.411738 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:05:24.411722    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"812b971dc18efc9edb045fd06fdeb7992fe6ef0d0483ecdd84b05c48940a691e\": container with ID starting with 812b971dc18efc9edb045fd06fdeb7992fe6ef0d0483ecdd84b05c48940a691e not found: ID does not exist" containerID="812b971dc18efc9edb045fd06fdeb7992fe6ef0d0483ecdd84b05c48940a691e"
Sep 16 18:05:24.411793 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.411747    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"812b971dc18efc9edb045fd06fdeb7992fe6ef0d0483ecdd84b05c48940a691e"} err="failed to get container status \"812b971dc18efc9edb045fd06fdeb7992fe6ef0d0483ecdd84b05c48940a691e\": rpc error: code = NotFound desc = could not find container \"812b971dc18efc9edb045fd06fdeb7992fe6ef0d0483ecdd84b05c48940a691e\": container with ID starting with 812b971dc18efc9edb045fd06fdeb7992fe6ef0d0483ecdd84b05c48940a691e not found: ID does not exist"
Sep 16 18:05:24.411793 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.411763    2581 scope.go:117] "RemoveContainer" containerID="373b450da6df74bb19894f8a4a5667ee096a6f2e60bbfe8db81e752f281c7b70"
Sep 16 18:05:24.412020 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:05:24.412003    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373b450da6df74bb19894f8a4a5667ee096a6f2e60bbfe8db81e752f281c7b70\": container with ID starting with 373b450da6df74bb19894f8a4a5667ee096a6f2e60bbfe8db81e752f281c7b70 not found: ID does not exist" containerID="373b450da6df74bb19894f8a4a5667ee096a6f2e60bbfe8db81e752f281c7b70"
Sep 16 18:05:24.412098 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.412028    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373b450da6df74bb19894f8a4a5667ee096a6f2e60bbfe8db81e752f281c7b70"} err="failed to get container status \"373b450da6df74bb19894f8a4a5667ee096a6f2e60bbfe8db81e752f281c7b70\": rpc error: code = NotFound desc = could not find container \"373b450da6df74bb19894f8a4a5667ee096a6f2e60bbfe8db81e752f281c7b70\": container with ID starting with 373b450da6df74bb19894f8a4a5667ee096a6f2e60bbfe8db81e752f281c7b70 not found: ID does not exist"
Sep 16 18:05:24.412098 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.412049    2581 scope.go:117] "RemoveContainer" containerID="e734704ac0fc5a362bc5c93fd2ee02f5e840bcda97cd20321c9b8ec537d91b7c"
Sep 16 18:05:24.412295 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:05:24.412277    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e734704ac0fc5a362bc5c93fd2ee02f5e840bcda97cd20321c9b8ec537d91b7c\": container with ID starting with e734704ac0fc5a362bc5c93fd2ee02f5e840bcda97cd20321c9b8ec537d91b7c not found: ID does not exist" containerID="e734704ac0fc5a362bc5c93fd2ee02f5e840bcda97cd20321c9b8ec537d91b7c"
Sep 16 18:05:24.412347 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.412299    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e734704ac0fc5a362bc5c93fd2ee02f5e840bcda97cd20321c9b8ec537d91b7c"} err="failed to get container status \"e734704ac0fc5a362bc5c93fd2ee02f5e840bcda97cd20321c9b8ec537d91b7c\": rpc error: code = NotFound desc = could not find container \"e734704ac0fc5a362bc5c93fd2ee02f5e840bcda97cd20321c9b8ec537d91b7c\": container with ID starting with e734704ac0fc5a362bc5c93fd2ee02f5e840bcda97cd20321c9b8ec537d91b7c not found: ID does not exist"
Sep 16 18:05:24.419251 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.419234    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mbp\" (UniqueName: \"kubernetes.io/projected/f270934c-6e6d-416f-87f7-700d9b34b259-kube-api-access-m8mbp\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-prfwv\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.435942 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.435920    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"]
Sep 16 18:05:24.461366 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.461347    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s6wrm"]
Sep 16 18:05:24.478029 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.478006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:24.757885 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:24.757814    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"]
Sep 16 18:05:24.761101 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:05:24.761073    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf270934c_6e6d_416f_87f7_700d9b34b259.slice/crio-f5ca933e6e30fbcc2cd2a916f8d5b139caaaf071e963bc2fccff3825fcc6f267 WatchSource:0}: Error finding container f5ca933e6e30fbcc2cd2a916f8d5b139caaaf071e963bc2fccff3825fcc6f267: Status 404 returned error can't find the container with id f5ca933e6e30fbcc2cd2a916f8d5b139caaaf071e963bc2fccff3825fcc6f267
Sep 16 18:05:25.396422 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:25.396387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" event={"ID":"f270934c-6e6d-416f-87f7-700d9b34b259","Type":"ContainerStarted","Data":"d8143ee7b6748318bf163682b600db4e70a89a4bc84314b09f26ec34f04577f1"}
Sep 16 18:05:25.396774 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:25.396458    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" event={"ID":"f270934c-6e6d-416f-87f7-700d9b34b259","Type":"ContainerStarted","Data":"f5ca933e6e30fbcc2cd2a916f8d5b139caaaf071e963bc2fccff3825fcc6f267"}
Sep 16 18:05:25.766656 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:25.766584    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="001af213-15f1-45d8-910b-63f67d62b2a3" path="/var/lib/kubelet/pods/001af213-15f1-45d8-910b-63f67d62b2a3/volumes"
Sep 16 18:05:29.414561 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:29.414532    2581 generic.go:356] "Generic (PLEG): container finished" podID="f270934c-6e6d-416f-87f7-700d9b34b259" containerID="d8143ee7b6748318bf163682b600db4e70a89a4bc84314b09f26ec34f04577f1" exitCode=0
Sep 16 18:05:29.414971 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:29.414602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" event={"ID":"f270934c-6e6d-416f-87f7-700d9b34b259","Type":"ContainerDied","Data":"d8143ee7b6748318bf163682b600db4e70a89a4bc84314b09f26ec34f04577f1"}
Sep 16 18:05:29.415843 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:29.415826    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:05:33.138368 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:33.138324    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 18:05:43.139124 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:43.139092    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:05:49.513488 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:49.513455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" event={"ID":"f270934c-6e6d-416f-87f7-700d9b34b259","Type":"ContainerStarted","Data":"d5448f9196ae5c682da00269650becc19521d89ae487a8d33dfc58a1c80a48a7"}
Sep 16 18:05:49.513950 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:49.513513    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" event={"ID":"f270934c-6e6d-416f-87f7-700d9b34b259","Type":"ContainerStarted","Data":"29a511559a04801c5ec97bb65a75b695b0350d6c45a25033cfa731eb3045885d"}
Sep 16 18:05:49.513950 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:49.513768    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:49.513950 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:49.513910    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:49.515077 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:49.515053    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 16 18:05:49.541719 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:49.541678    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" podStartSLOduration=2.712405872 podStartE2EDuration="25.541665558s" totalImagesPullingTime="19.19347519s" totalInitContainerRuntime="3.635784496s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:05:49.539217074 +0000 UTC m=+2478.327839606" watchObservedRunningTime="2026-09-16 18:05:49.541665558 +0000 UTC m=+2478.330288091"
Sep 16 18:05:50.517844 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:50.517805    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 16 18:05:51.073567 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:51.073532    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:05:51.169320 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:51.169288    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:05:51.185728 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:51.185701    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"]
Sep 16 18:05:51.185968 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:51.185945    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kserve-container" containerID="cri-o://7080ec9de0638dc331344dc5a906bcbba177aa801e1462811f01c6f7dbef5742" gracePeriod=30
Sep 16 18:05:51.186056 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:51.185963    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kube-rbac-proxy" containerID="cri-o://60188014cd0cacb276acce39260643db578097933537de12833d92dc4744e508" gracePeriod=30
Sep 16 18:05:51.524386 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:51.524358    2581 generic.go:356] "Generic (PLEG): container finished" podID="bd2b579d-d721-4ede-a501-99f75da66cde" containerID="60188014cd0cacb276acce39260643db578097933537de12833d92dc4744e508" exitCode=2
Sep 16 18:05:51.524706 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:51.524433    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" event={"ID":"bd2b579d-d721-4ede-a501-99f75da66cde","Type":"ContainerDied","Data":"60188014cd0cacb276acce39260643db578097933537de12833d92dc4744e508"}
Sep 16 18:05:51.525493 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:51.525017    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 16 18:05:53.133408 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:53.133366    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Sep 16 18:05:53.137717 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:53.137696    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 18:05:54.922246 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:54.922222    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:05:55.022146 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.022126    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdcz6\" (UniqueName: \"kubernetes.io/projected/bd2b579d-d721-4ede-a501-99f75da66cde-kube-api-access-fdcz6\") pod \"bd2b579d-d721-4ede-a501-99f75da66cde\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") "
Sep 16 18:05:55.022291 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.022196    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd2b579d-d721-4ede-a501-99f75da66cde-kube-rbac-proxy-sar-config\") pod \"bd2b579d-d721-4ede-a501-99f75da66cde\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") "
Sep 16 18:05:55.022291 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.022252    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd2b579d-d721-4ede-a501-99f75da66cde-proxy-tls\") pod \"bd2b579d-d721-4ede-a501-99f75da66cde\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") "
Sep 16 18:05:55.022291 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.022274    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd2b579d-d721-4ede-a501-99f75da66cde-cabundle-cert\") pod \"bd2b579d-d721-4ede-a501-99f75da66cde\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") "
Sep 16 18:05:55.022452 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.022312    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd2b579d-d721-4ede-a501-99f75da66cde-kserve-provision-location\") pod \"bd2b579d-d721-4ede-a501-99f75da66cde\" (UID: \"bd2b579d-d721-4ede-a501-99f75da66cde\") "
Sep 16 18:05:55.022628 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.022552    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2b579d-d721-4ede-a501-99f75da66cde-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd2b579d-d721-4ede-a501-99f75da66cde" (UID: "bd2b579d-d721-4ede-a501-99f75da66cde"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:05:55.022691 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.022662    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2b579d-d721-4ede-a501-99f75da66cde-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd2b579d-d721-4ede-a501-99f75da66cde" (UID: "bd2b579d-d721-4ede-a501-99f75da66cde"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:05:55.022691 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.022662    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2b579d-d721-4ede-a501-99f75da66cde-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd2b579d-d721-4ede-a501-99f75da66cde" (UID: "bd2b579d-d721-4ede-a501-99f75da66cde"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:05:55.024387 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.024367    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2b579d-d721-4ede-a501-99f75da66cde-kube-api-access-fdcz6" (OuterVolumeSpecName: "kube-api-access-fdcz6") pod "bd2b579d-d721-4ede-a501-99f75da66cde" (UID: "bd2b579d-d721-4ede-a501-99f75da66cde"). InnerVolumeSpecName "kube-api-access-fdcz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:05:55.024387 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.024373    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd2b579d-d721-4ede-a501-99f75da66cde-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd2b579d-d721-4ede-a501-99f75da66cde" (UID: "bd2b579d-d721-4ede-a501-99f75da66cde"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:05:55.122816 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.122795    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdcz6\" (UniqueName: \"kubernetes.io/projected/bd2b579d-d721-4ede-a501-99f75da66cde-kube-api-access-fdcz6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:05:55.122816 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.122815    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd2b579d-d721-4ede-a501-99f75da66cde-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:05:55.122958 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.122824    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd2b579d-d721-4ede-a501-99f75da66cde-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:05:55.122958 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.122834    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd2b579d-d721-4ede-a501-99f75da66cde-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:05:55.122958 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.122841    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd2b579d-d721-4ede-a501-99f75da66cde-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:05:55.543358 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.543324    2581 generic.go:356] "Generic (PLEG): container finished" podID="bd2b579d-d721-4ede-a501-99f75da66cde" containerID="7080ec9de0638dc331344dc5a906bcbba177aa801e1462811f01c6f7dbef5742" exitCode=0
Sep 16 18:05:55.543491 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.543383    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"
Sep 16 18:05:55.543491 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.543394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" event={"ID":"bd2b579d-d721-4ede-a501-99f75da66cde","Type":"ContainerDied","Data":"7080ec9de0638dc331344dc5a906bcbba177aa801e1462811f01c6f7dbef5742"}
Sep 16 18:05:55.543491 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.543468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv" event={"ID":"bd2b579d-d721-4ede-a501-99f75da66cde","Type":"ContainerDied","Data":"eddadad5f76b3a435dc42f1e17d3f833263f387689df57b023877584f114df3d"}
Sep 16 18:05:55.543620 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.543495    2581 scope.go:117] "RemoveContainer" containerID="60188014cd0cacb276acce39260643db578097933537de12833d92dc4744e508"
Sep 16 18:05:55.552227 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.552210    2581 scope.go:117] "RemoveContainer" containerID="7080ec9de0638dc331344dc5a906bcbba177aa801e1462811f01c6f7dbef5742"
Sep 16 18:05:55.559463 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.559432    2581 scope.go:117] "RemoveContainer" containerID="3a600556e978b6b65aa671f86e84c7d0a6aae939203ea1b7f9f3818dcecf530a"
Sep 16 18:05:55.566546 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.566526    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"]
Sep 16 18:05:55.567309 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.567295    2581 scope.go:117] "RemoveContainer" containerID="60188014cd0cacb276acce39260643db578097933537de12833d92dc4744e508"
Sep 16 18:05:55.567559 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:05:55.567543    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60188014cd0cacb276acce39260643db578097933537de12833d92dc4744e508\": container with ID starting with 60188014cd0cacb276acce39260643db578097933537de12833d92dc4744e508 not found: ID does not exist" containerID="60188014cd0cacb276acce39260643db578097933537de12833d92dc4744e508"
Sep 16 18:05:55.567604 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.567567    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60188014cd0cacb276acce39260643db578097933537de12833d92dc4744e508"} err="failed to get container status \"60188014cd0cacb276acce39260643db578097933537de12833d92dc4744e508\": rpc error: code = NotFound desc = could not find container \"60188014cd0cacb276acce39260643db578097933537de12833d92dc4744e508\": container with ID starting with 60188014cd0cacb276acce39260643db578097933537de12833d92dc4744e508 not found: ID does not exist"
Sep 16 18:05:55.567604 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.567583    2581 scope.go:117] "RemoveContainer" containerID="7080ec9de0638dc331344dc5a906bcbba177aa801e1462811f01c6f7dbef5742"
Sep 16 18:05:55.567866 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:05:55.567838    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7080ec9de0638dc331344dc5a906bcbba177aa801e1462811f01c6f7dbef5742\": container with ID starting with 7080ec9de0638dc331344dc5a906bcbba177aa801e1462811f01c6f7dbef5742 not found: ID does not exist" containerID="7080ec9de0638dc331344dc5a906bcbba177aa801e1462811f01c6f7dbef5742"
Sep 16 18:05:55.568002 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.567880    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7080ec9de0638dc331344dc5a906bcbba177aa801e1462811f01c6f7dbef5742"} err="failed to get container status \"7080ec9de0638dc331344dc5a906bcbba177aa801e1462811f01c6f7dbef5742\": rpc error: code = NotFound desc = could not find container \"7080ec9de0638dc331344dc5a906bcbba177aa801e1462811f01c6f7dbef5742\": container with ID starting with 7080ec9de0638dc331344dc5a906bcbba177aa801e1462811f01c6f7dbef5742 not found: ID does not exist"
Sep 16 18:05:55.568002 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.567919    2581 scope.go:117] "RemoveContainer" containerID="3a600556e978b6b65aa671f86e84c7d0a6aae939203ea1b7f9f3818dcecf530a"
Sep 16 18:05:55.568213 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:05:55.568196    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a600556e978b6b65aa671f86e84c7d0a6aae939203ea1b7f9f3818dcecf530a\": container with ID starting with 3a600556e978b6b65aa671f86e84c7d0a6aae939203ea1b7f9f3818dcecf530a not found: ID does not exist" containerID="3a600556e978b6b65aa671f86e84c7d0a6aae939203ea1b7f9f3818dcecf530a"
Sep 16 18:05:55.568273 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.568218    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a600556e978b6b65aa671f86e84c7d0a6aae939203ea1b7f9f3818dcecf530a"} err="failed to get container status \"3a600556e978b6b65aa671f86e84c7d0a6aae939203ea1b7f9f3818dcecf530a\": rpc error: code = NotFound desc = could not find container \"3a600556e978b6b65aa671f86e84c7d0a6aae939203ea1b7f9f3818dcecf530a\": container with ID starting with 3a600556e978b6b65aa671f86e84c7d0a6aae939203ea1b7f9f3818dcecf530a not found: ID does not exist"
Sep 16 18:05:55.570516 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.570497    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-b5wmv"]
Sep 16 18:05:55.765672 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:55.765652    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" path="/var/lib/kubelet/pods/bd2b579d-d721-4ede-a501-99f75da66cde/volumes"
Sep 16 18:05:56.529410 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:56.529388    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:05:56.529903 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:05:56.529867    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 16 18:06:01.291869 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:01.291830    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:01.434309 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:01.434277    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:06.530016 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:06.529974    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 16 18:06:16.530669 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:16.530622    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 16 18:06:26.530791 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:26.530752    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 16 18:06:36.218630 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:36.218597    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:36.530495 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:36.530466    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 16 18:06:46.530793 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:46.530767    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:06:54.130641 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:54.130606    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"]
Sep 16 18:06:54.131151 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:54.130881    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kserve-container" containerID="cri-o://29a511559a04801c5ec97bb65a75b695b0350d6c45a25033cfa731eb3045885d" gracePeriod=30
Sep 16 18:06:54.131151 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:54.130911    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kube-rbac-proxy" containerID="cri-o://d5448f9196ae5c682da00269650becc19521d89ae487a8d33dfc58a1c80a48a7" gracePeriod=30
Sep 16 18:06:54.157928 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:54.157858    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:54.787409 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:54.787379    2581 generic.go:356] "Generic (PLEG): container finished" podID="f270934c-6e6d-416f-87f7-700d9b34b259" containerID="d5448f9196ae5c682da00269650becc19521d89ae487a8d33dfc58a1c80a48a7" exitCode=2
Sep 16 18:06:54.787576 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:54.787449    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" event={"ID":"f270934c-6e6d-416f-87f7-700d9b34b259","Type":"ContainerDied","Data":"d5448f9196ae5c682da00269650becc19521d89ae487a8d33dfc58a1c80a48a7"}
Sep 16 18:06:56.524908 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.524856    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 16 18:06:56.530141 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.530120    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 16 18:06:56.798598 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.798568    2581 generic.go:356] "Generic (PLEG): container finished" podID="f270934c-6e6d-416f-87f7-700d9b34b259" containerID="29a511559a04801c5ec97bb65a75b695b0350d6c45a25033cfa731eb3045885d" exitCode=0
Sep 16 18:06:56.798728 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.798641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" event={"ID":"f270934c-6e6d-416f-87f7-700d9b34b259","Type":"ContainerDied","Data":"29a511559a04801c5ec97bb65a75b695b0350d6c45a25033cfa731eb3045885d"}
Sep 16 18:06:56.877421 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.877397    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:06:56.944376 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.944350    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8mbp\" (UniqueName: \"kubernetes.io/projected/f270934c-6e6d-416f-87f7-700d9b34b259-kube-api-access-m8mbp\") pod \"f270934c-6e6d-416f-87f7-700d9b34b259\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") "
Sep 16 18:06:56.944499 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.944395    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f270934c-6e6d-416f-87f7-700d9b34b259-kserve-provision-location\") pod \"f270934c-6e6d-416f-87f7-700d9b34b259\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") "
Sep 16 18:06:56.944499 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.944443    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f270934c-6e6d-416f-87f7-700d9b34b259-kube-rbac-proxy-sar-config\") pod \"f270934c-6e6d-416f-87f7-700d9b34b259\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") "
Sep 16 18:06:56.944618 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.944503    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f270934c-6e6d-416f-87f7-700d9b34b259-proxy-tls\") pod \"f270934c-6e6d-416f-87f7-700d9b34b259\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") "
Sep 16 18:06:56.944674 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.944612    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f270934c-6e6d-416f-87f7-700d9b34b259-cabundle-cert\") pod \"f270934c-6e6d-416f-87f7-700d9b34b259\" (UID: \"f270934c-6e6d-416f-87f7-700d9b34b259\") "
Sep 16 18:06:56.944823 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.944737    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f270934c-6e6d-416f-87f7-700d9b34b259-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f270934c-6e6d-416f-87f7-700d9b34b259" (UID: "f270934c-6e6d-416f-87f7-700d9b34b259"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:06:56.944911 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.944838    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f270934c-6e6d-416f-87f7-700d9b34b259-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f270934c-6e6d-416f-87f7-700d9b34b259" (UID: "f270934c-6e6d-416f-87f7-700d9b34b259"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:06:56.944911 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.944863    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f270934c-6e6d-416f-87f7-700d9b34b259-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:06:56.945002 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.944916    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f270934c-6e6d-416f-87f7-700d9b34b259-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f270934c-6e6d-416f-87f7-700d9b34b259" (UID: "f270934c-6e6d-416f-87f7-700d9b34b259"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:06:56.946470 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.946447    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f270934c-6e6d-416f-87f7-700d9b34b259-kube-api-access-m8mbp" (OuterVolumeSpecName: "kube-api-access-m8mbp") pod "f270934c-6e6d-416f-87f7-700d9b34b259" (UID: "f270934c-6e6d-416f-87f7-700d9b34b259"). InnerVolumeSpecName "kube-api-access-m8mbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:06:56.946562 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:56.946544    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f270934c-6e6d-416f-87f7-700d9b34b259-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f270934c-6e6d-416f-87f7-700d9b34b259" (UID: "f270934c-6e6d-416f-87f7-700d9b34b259"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:06:57.045462 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:57.045437    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f270934c-6e6d-416f-87f7-700d9b34b259-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:06:57.045462 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:57.045460    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f270934c-6e6d-416f-87f7-700d9b34b259-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:06:57.045595 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:57.045473    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8mbp\" (UniqueName: \"kubernetes.io/projected/f270934c-6e6d-416f-87f7-700d9b34b259-kube-api-access-m8mbp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:06:57.045595 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:57.045486    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f270934c-6e6d-416f-87f7-700d9b34b259-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:06:57.804214 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:57.804183    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv" event={"ID":"f270934c-6e6d-416f-87f7-700d9b34b259","Type":"ContainerDied","Data":"f5ca933e6e30fbcc2cd2a916f8d5b139caaaf071e963bc2fccff3825fcc6f267"}
Sep 16 18:06:57.804214 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:57.804203    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"
Sep 16 18:06:57.804597 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:57.804242    2581 scope.go:117] "RemoveContainer" containerID="d5448f9196ae5c682da00269650becc19521d89ae487a8d33dfc58a1c80a48a7"
Sep 16 18:06:57.812344 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:57.812325    2581 scope.go:117] "RemoveContainer" containerID="29a511559a04801c5ec97bb65a75b695b0350d6c45a25033cfa731eb3045885d"
Sep 16 18:06:57.819485 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:57.819467    2581 scope.go:117] "RemoveContainer" containerID="d8143ee7b6748318bf163682b600db4e70a89a4bc84314b09f26ec34f04577f1"
Sep 16 18:06:57.830311 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:57.830292    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"]
Sep 16 18:06:57.838159 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:57.838140    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-prfwv"]
Sep 16 18:06:59.397265 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.397237    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:59.469411 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469381    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"]
Sep 16 18:06:59.469699 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469686    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kube-rbac-proxy"
Sep 16 18:06:59.469742 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469702    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kube-rbac-proxy"
Sep 16 18:06:59.469742 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469711    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kserve-container"
Sep 16 18:06:59.469742 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469717    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kserve-container"
Sep 16 18:06:59.469742 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469723    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kserve-container"
Sep 16 18:06:59.469742 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469729    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kserve-container"
Sep 16 18:06:59.469907 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469749    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="storage-initializer"
Sep 16 18:06:59.469907 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469756    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="storage-initializer"
Sep 16 18:06:59.469907 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469763    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kube-rbac-proxy"
Sep 16 18:06:59.469907 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469768    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kube-rbac-proxy"
Sep 16 18:06:59.469907 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469774    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="storage-initializer"
Sep 16 18:06:59.469907 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469779    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="storage-initializer"
Sep 16 18:06:59.469907 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469839    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kube-rbac-proxy"
Sep 16 18:06:59.469907 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469846    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kserve-container"
Sep 16 18:06:59.469907 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469851    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" containerName="kube-rbac-proxy"
Sep 16 18:06:59.469907 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.469858    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd2b579d-d721-4ede-a501-99f75da66cde" containerName="kserve-container"
Sep 16 18:06:59.474514 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.474497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.481063 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.481041    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 16 18:06:59.481181 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.481059    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 18:06:59.481254 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.481224    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:06:59.481366 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.481351    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\""
Sep 16 18:06:59.482269 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.482254    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:06:59.482337 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.482318    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:06:59.482379 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.482363    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:06:59.497968 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.497941    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"]
Sep 16 18:06:59.562062 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.562042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba195060-e266-48fb-b7be-f7a11b4634a0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.562239 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.562077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba195060-e266-48fb-b7be-f7a11b4634a0-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.562239 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.562102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmf47\" (UniqueName: \"kubernetes.io/projected/ba195060-e266-48fb-b7be-f7a11b4634a0-kube-api-access-kmf47\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.562239 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.562176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba195060-e266-48fb-b7be-f7a11b4634a0-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.562239 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.562200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba195060-e266-48fb-b7be-f7a11b4634a0-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.567183 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.567165    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:59.662489 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.662438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba195060-e266-48fb-b7be-f7a11b4634a0-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.662489 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.662467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmf47\" (UniqueName: \"kubernetes.io/projected/ba195060-e266-48fb-b7be-f7a11b4634a0-kube-api-access-kmf47\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.662598 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.662497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba195060-e266-48fb-b7be-f7a11b4634a0-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.662598 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.662512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba195060-e266-48fb-b7be-f7a11b4634a0-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.662598 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.662560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba195060-e266-48fb-b7be-f7a11b4634a0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.662928 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.662885    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba195060-e266-48fb-b7be-f7a11b4634a0-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.663127 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.663108    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba195060-e266-48fb-b7be-f7a11b4634a0-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.663223 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.663208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba195060-e266-48fb-b7be-f7a11b4634a0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.665072 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.665053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba195060-e266-48fb-b7be-f7a11b4634a0-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.673354 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.673324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmf47\" (UniqueName: \"kubernetes.io/projected/ba195060-e266-48fb-b7be-f7a11b4634a0-kube-api-access-kmf47\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:06:59.766043 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.766020    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f270934c-6e6d-416f-87f7-700d9b34b259" path="/var/lib/kubelet/pods/f270934c-6e6d-416f-87f7-700d9b34b259/volumes"
Sep 16 18:06:59.783741 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:06:59.783726    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:07:00.062332 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:00.062309    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"]
Sep 16 18:07:00.063830 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:07:00.063798    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba195060_e266_48fb_b7be_f7a11b4634a0.slice/crio-724bea5956f44341989217684d92f456e8da3f71a28a6ff43ed0f76cb1c7a49c WatchSource:0}: Error finding container 724bea5956f44341989217684d92f456e8da3f71a28a6ff43ed0f76cb1c7a49c: Status 404 returned error can't find the container with id 724bea5956f44341989217684d92f456e8da3f71a28a6ff43ed0f76cb1c7a49c
Sep 16 18:07:00.819546 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:00.819515    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8" event={"ID":"ba195060-e266-48fb-b7be-f7a11b4634a0","Type":"ContainerStarted","Data":"c0388db32e85550181ca13289901a10bd7119e496bd507078d4815d1ee031797"}
Sep 16 18:07:00.819909 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:00.819574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8" event={"ID":"ba195060-e266-48fb-b7be-f7a11b4634a0","Type":"ContainerStarted","Data":"724bea5956f44341989217684d92f456e8da3f71a28a6ff43ed0f76cb1c7a49c"}
Sep 16 18:07:04.836946 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:04.836920    2581 generic.go:356] "Generic (PLEG): container finished" podID="ba195060-e266-48fb-b7be-f7a11b4634a0" containerID="c0388db32e85550181ca13289901a10bd7119e496bd507078d4815d1ee031797" exitCode=0
Sep 16 18:07:04.837268 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:04.836994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8" event={"ID":"ba195060-e266-48fb-b7be-f7a11b4634a0","Type":"ContainerDied","Data":"c0388db32e85550181ca13289901a10bd7119e496bd507078d4815d1ee031797"}
Sep 16 18:07:05.842379 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:05.842349    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8" event={"ID":"ba195060-e266-48fb-b7be-f7a11b4634a0","Type":"ContainerStarted","Data":"8d615ae55779780e8170d0bd2d944b053d67459147d5d658b66596892be908ee"}
Sep 16 18:07:05.842720 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:05.842398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8" event={"ID":"ba195060-e266-48fb-b7be-f7a11b4634a0","Type":"ContainerStarted","Data":"9ba172d02fd5ef96570998426668fc204afcdf77a29c46cf724fe23968f11a5d"}
Sep 16 18:07:05.842720 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:05.842673    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:07:05.842720 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:05.842718    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:07:05.867199 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:05.867148    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8" podStartSLOduration=2.999633229 podStartE2EDuration="6.867132347s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.867499118s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:07:05.865651355 +0000 UTC m=+2554.654273926" watchObservedRunningTime="2026-09-16 18:07:05.867132347 +0000 UTC m=+2554.655754879"
Sep 16 18:07:11.853621 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:11.853588    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:07:31.463274 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:31.463245    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:07:36.595563 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:36.595529    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:07:36.878102 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:36.878038    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:07:41.859942 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:41.859877    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8" podUID="ba195060-e266-48fb-b7be-f7a11b4634a0" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 18:07:51.856457 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:51.856421    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:07:59.412983 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:59.412871    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"]
Sep 16 18:07:59.413474 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:59.413159    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8" podUID="ba195060-e266-48fb-b7be-f7a11b4634a0" containerName="kserve-container" containerID="cri-o://9ba172d02fd5ef96570998426668fc204afcdf77a29c46cf724fe23968f11a5d" gracePeriod=30
Sep 16 18:07:59.413474 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:59.413238    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8" podUID="ba195060-e266-48fb-b7be-f7a11b4634a0" containerName="kube-rbac-proxy" containerID="cri-o://8d615ae55779780e8170d0bd2d944b053d67459147d5d658b66596892be908ee" gracePeriod=30
Sep 16 18:07:59.427689 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:07:59.427662    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:08:00.051654 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:00.051625    2581 generic.go:356] "Generic (PLEG): container finished" podID="ba195060-e266-48fb-b7be-f7a11b4634a0" containerID="8d615ae55779780e8170d0bd2d944b053d67459147d5d658b66596892be908ee" exitCode=2
Sep 16 18:08:00.051822 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:00.051686    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8" event={"ID":"ba195060-e266-48fb-b7be-f7a11b4634a0","Type":"ContainerDied","Data":"8d615ae55779780e8170d0bd2d944b053d67459147d5d658b66596892be908ee"}
Sep 16 18:08:01.848267 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:01.848229    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8" podUID="ba195060-e266-48fb-b7be-f7a11b4634a0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Sep 16 18:08:04.570681 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.570646    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:08:04.621731 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.621706    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"]
Sep 16 18:08:04.625714 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.625695    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:04.628986 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.628943    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 16 18:08:04.629095 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.629077    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 16 18:08:04.643320 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.643298    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"]
Sep 16 18:08:04.711585 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.711559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:04.711726 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.711599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:04.711726 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.711620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:04.711726 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.711637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqsbn\" (UniqueName: \"kubernetes.io/projected/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kube-api-access-gqsbn\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:04.711934 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.711755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:04.759510 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.759489    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:08:04.812926 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.812855    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba195060-e266-48fb-b7be-f7a11b4634a0-kserve-provision-location\") pod \"ba195060-e266-48fb-b7be-f7a11b4634a0\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") "
Sep 16 18:08:04.813032 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.812921    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmf47\" (UniqueName: \"kubernetes.io/projected/ba195060-e266-48fb-b7be-f7a11b4634a0-kube-api-access-kmf47\") pod \"ba195060-e266-48fb-b7be-f7a11b4634a0\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") "
Sep 16 18:08:04.813032 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.812954    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba195060-e266-48fb-b7be-f7a11b4634a0-cabundle-cert\") pod \"ba195060-e266-48fb-b7be-f7a11b4634a0\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") "
Sep 16 18:08:04.813032 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.812983    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba195060-e266-48fb-b7be-f7a11b4634a0-proxy-tls\") pod \"ba195060-e266-48fb-b7be-f7a11b4634a0\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") "
Sep 16 18:08:04.813032 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.813008    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba195060-e266-48fb-b7be-f7a11b4634a0-kube-rbac-proxy-sar-config\") pod \"ba195060-e266-48fb-b7be-f7a11b4634a0\" (UID: \"ba195060-e266-48fb-b7be-f7a11b4634a0\") "
Sep 16 18:08:04.813245 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.813119    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba195060-e266-48fb-b7be-f7a11b4634a0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ba195060-e266-48fb-b7be-f7a11b4634a0" (UID: "ba195060-e266-48fb-b7be-f7a11b4634a0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:04.813245 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.813124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:04.813355 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.813268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:04.813355 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.813315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:04.813355 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.813335    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba195060-e266-48fb-b7be-f7a11b4634a0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ba195060-e266-48fb-b7be-f7a11b4634a0" (UID: "ba195060-e266-48fb-b7be-f7a11b4634a0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:08:04.813355 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.813344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:04.813554 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.813355    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba195060-e266-48fb-b7be-f7a11b4634a0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ba195060-e266-48fb-b7be-f7a11b4634a0" (UID: "ba195060-e266-48fb-b7be-f7a11b4634a0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:08:04.813554 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.813373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqsbn\" (UniqueName: \"kubernetes.io/projected/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kube-api-access-gqsbn\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:04.813554 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.813458    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba195060-e266-48fb-b7be-f7a11b4634a0-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:08:04.813554 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.813475    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba195060-e266-48fb-b7be-f7a11b4634a0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:08:04.813554 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.813492    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba195060-e266-48fb-b7be-f7a11b4634a0-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:08:04.813554 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:08:04.813459    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 16 18:08:04.813554 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:08:04.813554    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-proxy-tls podName:e64b1f1a-5f48-42ce-a5fc-94a9110fd75a nodeName:}" failed. No retries permitted until 2026-09-16 18:08:05.313536597 +0000 UTC m=+2614.102159128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-zphw6" (UID: "e64b1f1a-5f48-42ce-a5fc-94a9110fd75a") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 16 18:08:04.814162 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.813648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:04.814162 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.813758    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:04.814162 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.813945    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:04.815242 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.815218    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba195060-e266-48fb-b7be-f7a11b4634a0-kube-api-access-kmf47" (OuterVolumeSpecName: "kube-api-access-kmf47") pod "ba195060-e266-48fb-b7be-f7a11b4634a0" (UID: "ba195060-e266-48fb-b7be-f7a11b4634a0"). InnerVolumeSpecName "kube-api-access-kmf47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:08:04.815457 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.815437    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba195060-e266-48fb-b7be-f7a11b4634a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ba195060-e266-48fb-b7be-f7a11b4634a0" (UID: "ba195060-e266-48fb-b7be-f7a11b4634a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:08:04.824875 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.824856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqsbn\" (UniqueName: \"kubernetes.io/projected/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kube-api-access-gqsbn\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:04.913963 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.913939    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmf47\" (UniqueName: \"kubernetes.io/projected/ba195060-e266-48fb-b7be-f7a11b4634a0-kube-api-access-kmf47\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:08:04.913963 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:04.913961    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba195060-e266-48fb-b7be-f7a11b4634a0-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:08:05.060702 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.060677    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:08:05.073066 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.073017    2581 generic.go:356] "Generic (PLEG): container finished" podID="ba195060-e266-48fb-b7be-f7a11b4634a0" containerID="9ba172d02fd5ef96570998426668fc204afcdf77a29c46cf724fe23968f11a5d" exitCode=0
Sep 16 18:08:05.073154 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.073076    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8" event={"ID":"ba195060-e266-48fb-b7be-f7a11b4634a0","Type":"ContainerDied","Data":"9ba172d02fd5ef96570998426668fc204afcdf77a29c46cf724fe23968f11a5d"}
Sep 16 18:08:05.073154 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.073117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8" event={"ID":"ba195060-e266-48fb-b7be-f7a11b4634a0","Type":"ContainerDied","Data":"724bea5956f44341989217684d92f456e8da3f71a28a6ff43ed0f76cb1c7a49c"}
Sep 16 18:08:05.073154 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.073141    2581 scope.go:117] "RemoveContainer" containerID="8d615ae55779780e8170d0bd2d944b053d67459147d5d658b66596892be908ee"
Sep 16 18:08:05.073263 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.073075    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"
Sep 16 18:08:05.081844 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.081827    2581 scope.go:117] "RemoveContainer" containerID="9ba172d02fd5ef96570998426668fc204afcdf77a29c46cf724fe23968f11a5d"
Sep 16 18:08:05.088961 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.088945    2581 scope.go:117] "RemoveContainer" containerID="c0388db32e85550181ca13289901a10bd7119e496bd507078d4815d1ee031797"
Sep 16 18:08:05.095740 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.095719    2581 scope.go:117] "RemoveContainer" containerID="8d615ae55779780e8170d0bd2d944b053d67459147d5d658b66596892be908ee"
Sep 16 18:08:05.096036 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:08:05.096015    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d615ae55779780e8170d0bd2d944b053d67459147d5d658b66596892be908ee\": container with ID starting with 8d615ae55779780e8170d0bd2d944b053d67459147d5d658b66596892be908ee not found: ID does not exist" containerID="8d615ae55779780e8170d0bd2d944b053d67459147d5d658b66596892be908ee"
Sep 16 18:08:05.096105 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.096042    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d615ae55779780e8170d0bd2d944b053d67459147d5d658b66596892be908ee"} err="failed to get container status \"8d615ae55779780e8170d0bd2d944b053d67459147d5d658b66596892be908ee\": rpc error: code = NotFound desc = could not find container \"8d615ae55779780e8170d0bd2d944b053d67459147d5d658b66596892be908ee\": container with ID starting with 8d615ae55779780e8170d0bd2d944b053d67459147d5d658b66596892be908ee not found: ID does not exist"
Sep 16 18:08:05.096105 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.096058    2581 scope.go:117] "RemoveContainer" containerID="9ba172d02fd5ef96570998426668fc204afcdf77a29c46cf724fe23968f11a5d"
Sep 16 18:08:05.096299 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:08:05.096274    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba172d02fd5ef96570998426668fc204afcdf77a29c46cf724fe23968f11a5d\": container with ID starting with 9ba172d02fd5ef96570998426668fc204afcdf77a29c46cf724fe23968f11a5d not found: ID does not exist" containerID="9ba172d02fd5ef96570998426668fc204afcdf77a29c46cf724fe23968f11a5d"
Sep 16 18:08:05.096339 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.096305    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba172d02fd5ef96570998426668fc204afcdf77a29c46cf724fe23968f11a5d"} err="failed to get container status \"9ba172d02fd5ef96570998426668fc204afcdf77a29c46cf724fe23968f11a5d\": rpc error: code = NotFound desc = could not find container \"9ba172d02fd5ef96570998426668fc204afcdf77a29c46cf724fe23968f11a5d\": container with ID starting with 9ba172d02fd5ef96570998426668fc204afcdf77a29c46cf724fe23968f11a5d not found: ID does not exist"
Sep 16 18:08:05.096339 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.096320    2581 scope.go:117] "RemoveContainer" containerID="c0388db32e85550181ca13289901a10bd7119e496bd507078d4815d1ee031797"
Sep 16 18:08:05.096548 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:08:05.096529    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0388db32e85550181ca13289901a10bd7119e496bd507078d4815d1ee031797\": container with ID starting with c0388db32e85550181ca13289901a10bd7119e496bd507078d4815d1ee031797 not found: ID does not exist" containerID="c0388db32e85550181ca13289901a10bd7119e496bd507078d4815d1ee031797"
Sep 16 18:08:05.096604 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.096557    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0388db32e85550181ca13289901a10bd7119e496bd507078d4815d1ee031797"} err="failed to get container status \"c0388db32e85550181ca13289901a10bd7119e496bd507078d4815d1ee031797\": rpc error: code = NotFound desc = could not find container \"c0388db32e85550181ca13289901a10bd7119e496bd507078d4815d1ee031797\": container with ID starting with c0388db32e85550181ca13289901a10bd7119e496bd507078d4815d1ee031797 not found: ID does not exist"
Sep 16 18:08:05.101351 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.101330    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"]
Sep 16 18:08:05.104381 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.104363    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-775t8"]
Sep 16 18:08:05.317363 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.317339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:05.319448 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.319430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zphw6\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:05.538705 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.538689    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:05.770507 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.768047    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba195060-e266-48fb-b7be-f7a11b4634a0" path="/var/lib/kubelet/pods/ba195060-e266-48fb-b7be-f7a11b4634a0/volumes"
Sep 16 18:08:05.817934 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:05.817913    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"]
Sep 16 18:08:05.819809 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:08:05.819769    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64b1f1a_5f48_42ce_a5fc_94a9110fd75a.slice/crio-580577bebf6923dd6ad63deb7ce4a8e6eb0abee1ee5f9eac67f7295f4731ac09 WatchSource:0}: Error finding container 580577bebf6923dd6ad63deb7ce4a8e6eb0abee1ee5f9eac67f7295f4731ac09: Status 404 returned error can't find the container with id 580577bebf6923dd6ad63deb7ce4a8e6eb0abee1ee5f9eac67f7295f4731ac09
Sep 16 18:08:06.080627 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:06.080551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" event={"ID":"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a","Type":"ContainerStarted","Data":"741f2566d6d3bdecf858111d98401060db994ecd86d5698b56c1bc0726edc5fe"}
Sep 16 18:08:06.080627 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:06.080605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" event={"ID":"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a","Type":"ContainerStarted","Data":"580577bebf6923dd6ad63deb7ce4a8e6eb0abee1ee5f9eac67f7295f4731ac09"}
Sep 16 18:08:10.097003 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:10.096975    2581 generic.go:356] "Generic (PLEG): container finished" podID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerID="741f2566d6d3bdecf858111d98401060db994ecd86d5698b56c1bc0726edc5fe" exitCode=0
Sep 16 18:08:10.097312 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:10.097010    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" event={"ID":"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a","Type":"ContainerDied","Data":"741f2566d6d3bdecf858111d98401060db994ecd86d5698b56c1bc0726edc5fe"}
Sep 16 18:08:11.103140 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:11.103110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" event={"ID":"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a","Type":"ContainerStarted","Data":"c0f98913f3be6f294b5cbdec2662c7df225ffcdf9b6acca394ccd8bd79f43825"}
Sep 16 18:08:11.103493 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:11.103186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" event={"ID":"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a","Type":"ContainerStarted","Data":"7c3f0624b6a3c1ff99e6e46b90bed059e7300474bb428bba43859c1634e900d6"}
Sep 16 18:08:11.103493 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:11.103362    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:11.129507 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:11.129465    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" podStartSLOduration=3.3276937 podStartE2EDuration="7.129451873s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.801758173s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:08:11.128343411 +0000 UTC m=+2619.916965944" watchObservedRunningTime="2026-09-16 18:08:11.129451873 +0000 UTC m=+2619.918074405"
Sep 16 18:08:12.107285 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:12.107257    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:12.108574 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:12.108548    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 18:08:13.111214 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:13.111179    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 18:08:18.115870 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:18.115839    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:08:18.116415 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:18.116392    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 18:08:28.117038 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:28.117004    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 18:08:29.389304 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:29.389273    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:08:38.116983 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:38.116943    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 18:08:48.116696 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:48.116661    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 18:08:58.116922 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:08:58.116864    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 18:09:06.948966 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:06.948931    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:08.116862 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:08.116832    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:09:12.187760 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.187715    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:12.276653 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.276622    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"]
Sep 16 18:09:12.277161 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.277144    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba195060-e266-48fb-b7be-f7a11b4634a0" containerName="kserve-container"
Sep 16 18:09:12.277225 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.277164    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba195060-e266-48fb-b7be-f7a11b4634a0" containerName="kserve-container"
Sep 16 18:09:12.277225 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.277178    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba195060-e266-48fb-b7be-f7a11b4634a0" containerName="storage-initializer"
Sep 16 18:09:12.277225 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.277184    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba195060-e266-48fb-b7be-f7a11b4634a0" containerName="storage-initializer"
Sep 16 18:09:12.277225 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.277199    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba195060-e266-48fb-b7be-f7a11b4634a0" containerName="kube-rbac-proxy"
Sep 16 18:09:12.277225 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.277205    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba195060-e266-48fb-b7be-f7a11b4634a0" containerName="kube-rbac-proxy"
Sep 16 18:09:12.277403 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.277253    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba195060-e266-48fb-b7be-f7a11b4634a0" containerName="kserve-container"
Sep 16 18:09:12.277403 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.277262    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba195060-e266-48fb-b7be-f7a11b4634a0" containerName="kube-rbac-proxy"
Sep 16 18:09:12.280188 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.280170    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.289356 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.289335    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 16 18:09:12.289560 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.289548    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 18:09:12.304430 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.304407    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"]
Sep 16 18:09:12.388303 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.388274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69461556-2d5d-4209-9468-df0cf9bd65ba-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.388435 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.388313    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69461556-2d5d-4209-9468-df0cf9bd65ba-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.388435 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.388337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmqp7\" (UniqueName: \"kubernetes.io/projected/69461556-2d5d-4209-9468-df0cf9bd65ba-kube-api-access-nmqp7\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.388435 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.388417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69461556-2d5d-4209-9468-df0cf9bd65ba-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.388598 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.388470    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69461556-2d5d-4209-9468-df0cf9bd65ba-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.438672 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.438611    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:12.489019 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.489000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69461556-2d5d-4209-9468-df0cf9bd65ba-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.489131 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.489033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69461556-2d5d-4209-9468-df0cf9bd65ba-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.489131 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.489056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmqp7\" (UniqueName: \"kubernetes.io/projected/69461556-2d5d-4209-9468-df0cf9bd65ba-kube-api-access-nmqp7\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.489229 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.489194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69461556-2d5d-4209-9468-df0cf9bd65ba-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.489277 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.489251    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69461556-2d5d-4209-9468-df0cf9bd65ba-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.489433 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.489413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69461556-2d5d-4209-9468-df0cf9bd65ba-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.489830 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.489814    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69461556-2d5d-4209-9468-df0cf9bd65ba-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.489876 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.489834    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69461556-2d5d-4209-9468-df0cf9bd65ba-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.491614 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.491596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69461556-2d5d-4209-9468-df0cf9bd65ba-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.505220 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.505196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmqp7\" (UniqueName: \"kubernetes.io/projected/69461556-2d5d-4209-9468-df0cf9bd65ba-kube-api-access-nmqp7\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.590449 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.590420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:12.869560 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:12.869532    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"]
Sep 16 18:09:12.872570 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:09:12.872544    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69461556_2d5d_4209_9468_df0cf9bd65ba.slice/crio-1fea6c125f8a6c2e6dc9b1478acf628eb43eac60c0f6175d894b054cd97826cf WatchSource:0}: Error finding container 1fea6c125f8a6c2e6dc9b1478acf628eb43eac60c0f6175d894b054cd97826cf: Status 404 returned error can't find the container with id 1fea6c125f8a6c2e6dc9b1478acf628eb43eac60c0f6175d894b054cd97826cf
Sep 16 18:09:13.358287 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:13.358251    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw" event={"ID":"69461556-2d5d-4209-9468-df0cf9bd65ba","Type":"ContainerStarted","Data":"dfc729f62ee525d56e5e90a5aed5aed9fafd168929e54240c1e7d3311bb8d887"}
Sep 16 18:09:13.358665 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:13.358308    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw" event={"ID":"69461556-2d5d-4209-9468-df0cf9bd65ba","Type":"ContainerStarted","Data":"1fea6c125f8a6c2e6dc9b1478acf628eb43eac60c0f6175d894b054cd97826cf"}
Sep 16 18:09:14.667729 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:14.667698    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"]
Sep 16 18:09:14.668132 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:14.667966    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="kserve-container" containerID="cri-o://7c3f0624b6a3c1ff99e6e46b90bed059e7300474bb428bba43859c1634e900d6" gracePeriod=30
Sep 16 18:09:14.668132 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:14.668046    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="kube-rbac-proxy" containerID="cri-o://c0f98913f3be6f294b5cbdec2662c7df225ffcdf9b6acca394ccd8bd79f43825" gracePeriod=30
Sep 16 18:09:14.677869 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:14.677824    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:15.369340 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:15.369298    2581 generic.go:356] "Generic (PLEG): container finished" podID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerID="c0f98913f3be6f294b5cbdec2662c7df225ffcdf9b6acca394ccd8bd79f43825" exitCode=2
Sep 16 18:09:15.369509 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:15.369368    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" event={"ID":"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a","Type":"ContainerDied","Data":"c0f98913f3be6f294b5cbdec2662c7df225ffcdf9b6acca394ccd8bd79f43825"}
Sep 16 18:09:17.379768 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:17.379734    2581 generic.go:356] "Generic (PLEG): container finished" podID="69461556-2d5d-4209-9468-df0cf9bd65ba" containerID="dfc729f62ee525d56e5e90a5aed5aed9fafd168929e54240c1e7d3311bb8d887" exitCode=0
Sep 16 18:09:17.380283 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:17.379813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw" event={"ID":"69461556-2d5d-4209-9468-df0cf9bd65ba","Type":"ContainerDied","Data":"dfc729f62ee525d56e5e90a5aed5aed9fafd168929e54240c1e7d3311bb8d887"}
Sep 16 18:09:18.011936 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.011884    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:09:18.139557 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.139472    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-proxy-tls\") pod \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") "
Sep 16 18:09:18.139557 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.139508    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-cabundle-cert\") pod \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") "
Sep 16 18:09:18.139557 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.139546    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kserve-provision-location\") pod \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") "
Sep 16 18:09:18.139840 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.139575    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kube-rbac-proxy-sar-config\") pod \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") "
Sep 16 18:09:18.139840 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.139593    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqsbn\" (UniqueName: \"kubernetes.io/projected/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kube-api-access-gqsbn\") pod \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\" (UID: \"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a\") "
Sep 16 18:09:18.139991 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.139918    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" (UID: "e64b1f1a-5f48-42ce-a5fc-94a9110fd75a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:09:18.139991 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.139983    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" (UID: "e64b1f1a-5f48-42ce-a5fc-94a9110fd75a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:09:18.140093 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.139992    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" (UID: "e64b1f1a-5f48-42ce-a5fc-94a9110fd75a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:09:18.141753 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.141728    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" (UID: "e64b1f1a-5f48-42ce-a5fc-94a9110fd75a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:09:18.141936 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.141904    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kube-api-access-gqsbn" (OuterVolumeSpecName: "kube-api-access-gqsbn") pod "e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" (UID: "e64b1f1a-5f48-42ce-a5fc-94a9110fd75a"). InnerVolumeSpecName "kube-api-access-gqsbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:09:18.240825 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.240783    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:09:18.240825 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.240811    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqsbn\" (UniqueName: \"kubernetes.io/projected/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kube-api-access-gqsbn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:09:18.240825 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.240820    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:09:18.240825 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.240830    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:09:18.240825 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.240838    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:09:18.386362 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.386329    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw" event={"ID":"69461556-2d5d-4209-9468-df0cf9bd65ba","Type":"ContainerStarted","Data":"3bf1459626b62d05ad7c4b7ed23c63c368e196dfd16743281d21d7fb38a34ac3"}
Sep 16 18:09:18.386805 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.386403    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw" event={"ID":"69461556-2d5d-4209-9468-df0cf9bd65ba","Type":"ContainerStarted","Data":"cc1758df32b1925d852d7271d636f4f06c40f2c939cd0ab2413658bce134e4f2"}
Sep 16 18:09:18.386805 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.386631    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:18.388435 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.388413    2581 generic.go:356] "Generic (PLEG): container finished" podID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerID="7c3f0624b6a3c1ff99e6e46b90bed059e7300474bb428bba43859c1634e900d6" exitCode=0
Sep 16 18:09:18.388571 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.388464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" event={"ID":"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a","Type":"ContainerDied","Data":"7c3f0624b6a3c1ff99e6e46b90bed059e7300474bb428bba43859c1634e900d6"}
Sep 16 18:09:18.388571 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.388470    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"
Sep 16 18:09:18.388571 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.388499    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6" event={"ID":"e64b1f1a-5f48-42ce-a5fc-94a9110fd75a","Type":"ContainerDied","Data":"580577bebf6923dd6ad63deb7ce4a8e6eb0abee1ee5f9eac67f7295f4731ac09"}
Sep 16 18:09:18.388571 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.388526    2581 scope.go:117] "RemoveContainer" containerID="c0f98913f3be6f294b5cbdec2662c7df225ffcdf9b6acca394ccd8bd79f43825"
Sep 16 18:09:18.397675 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.397658    2581 scope.go:117] "RemoveContainer" containerID="7c3f0624b6a3c1ff99e6e46b90bed059e7300474bb428bba43859c1634e900d6"
Sep 16 18:09:18.405559 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.405538    2581 scope.go:117] "RemoveContainer" containerID="741f2566d6d3bdecf858111d98401060db994ecd86d5698b56c1bc0726edc5fe"
Sep 16 18:09:18.413421 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.413399    2581 scope.go:117] "RemoveContainer" containerID="c0f98913f3be6f294b5cbdec2662c7df225ffcdf9b6acca394ccd8bd79f43825"
Sep 16 18:09:18.413746 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:09:18.413727    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f98913f3be6f294b5cbdec2662c7df225ffcdf9b6acca394ccd8bd79f43825\": container with ID starting with c0f98913f3be6f294b5cbdec2662c7df225ffcdf9b6acca394ccd8bd79f43825 not found: ID does not exist" containerID="c0f98913f3be6f294b5cbdec2662c7df225ffcdf9b6acca394ccd8bd79f43825"
Sep 16 18:09:18.413797 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.413756    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f98913f3be6f294b5cbdec2662c7df225ffcdf9b6acca394ccd8bd79f43825"} err="failed to get container status \"c0f98913f3be6f294b5cbdec2662c7df225ffcdf9b6acca394ccd8bd79f43825\": rpc error: code = NotFound desc = could not find container \"c0f98913f3be6f294b5cbdec2662c7df225ffcdf9b6acca394ccd8bd79f43825\": container with ID starting with c0f98913f3be6f294b5cbdec2662c7df225ffcdf9b6acca394ccd8bd79f43825 not found: ID does not exist"
Sep 16 18:09:18.413797 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.413774    2581 scope.go:117] "RemoveContainer" containerID="7c3f0624b6a3c1ff99e6e46b90bed059e7300474bb428bba43859c1634e900d6"
Sep 16 18:09:18.414092 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:09:18.414073    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3f0624b6a3c1ff99e6e46b90bed059e7300474bb428bba43859c1634e900d6\": container with ID starting with 7c3f0624b6a3c1ff99e6e46b90bed059e7300474bb428bba43859c1634e900d6 not found: ID does not exist" containerID="7c3f0624b6a3c1ff99e6e46b90bed059e7300474bb428bba43859c1634e900d6"
Sep 16 18:09:18.414148 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.414098    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3f0624b6a3c1ff99e6e46b90bed059e7300474bb428bba43859c1634e900d6"} err="failed to get container status \"7c3f0624b6a3c1ff99e6e46b90bed059e7300474bb428bba43859c1634e900d6\": rpc error: code = NotFound desc = could not find container \"7c3f0624b6a3c1ff99e6e46b90bed059e7300474bb428bba43859c1634e900d6\": container with ID starting with 7c3f0624b6a3c1ff99e6e46b90bed059e7300474bb428bba43859c1634e900d6 not found: ID does not exist"
Sep 16 18:09:18.414148 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.414119    2581 scope.go:117] "RemoveContainer" containerID="741f2566d6d3bdecf858111d98401060db994ecd86d5698b56c1bc0726edc5fe"
Sep 16 18:09:18.414387 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:09:18.414363    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"741f2566d6d3bdecf858111d98401060db994ecd86d5698b56c1bc0726edc5fe\": container with ID starting with 741f2566d6d3bdecf858111d98401060db994ecd86d5698b56c1bc0726edc5fe not found: ID does not exist" containerID="741f2566d6d3bdecf858111d98401060db994ecd86d5698b56c1bc0726edc5fe"
Sep 16 18:09:18.414442 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.414393    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741f2566d6d3bdecf858111d98401060db994ecd86d5698b56c1bc0726edc5fe"} err="failed to get container status \"741f2566d6d3bdecf858111d98401060db994ecd86d5698b56c1bc0726edc5fe\": rpc error: code = NotFound desc = could not find container \"741f2566d6d3bdecf858111d98401060db994ecd86d5698b56c1bc0726edc5fe\": container with ID starting with 741f2566d6d3bdecf858111d98401060db994ecd86d5698b56c1bc0726edc5fe not found: ID does not exist"
Sep 16 18:09:18.460816 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.460762    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw" podStartSLOduration=2.660255176 podStartE2EDuration="6.460744804s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.800489628s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:09:18.434665742 +0000 UTC m=+2687.223288276" watchObservedRunningTime="2026-09-16 18:09:18.460744804 +0000 UTC m=+2687.249367337"
Sep 16 18:09:18.461240 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.461225    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"]
Sep 16 18:09:18.465152 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:18.465127    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zphw6"]
Sep 16 18:09:19.394982 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:19.394947    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:19.767086 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:19.766999    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" path="/var/lib/kubelet/pods/e64b1f1a-5f48-42ce-a5fc-94a9110fd75a/volumes"
Sep 16 18:09:19.867415 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:19.867372    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:20.237471 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:20.237432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:25.405962 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:25.405873    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:09:31.892023 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:31.891992    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 18:09:31.893756 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:31.893737    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 18:09:55.409655 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:09:55.409622    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:10:02.318984 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:02.318952    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"]
Sep 16 18:10:02.319477 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:02.319180    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw" podUID="69461556-2d5d-4209-9468-df0cf9bd65ba" containerName="kserve-container" containerID="cri-o://cc1758df32b1925d852d7271d636f4f06c40f2c939cd0ab2413658bce134e4f2" gracePeriod=30
Sep 16 18:10:02.319477 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:02.319263    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw" podUID="69461556-2d5d-4209-9468-df0cf9bd65ba" containerName="kube-rbac-proxy" containerID="cri-o://3bf1459626b62d05ad7c4b7ed23c63c368e196dfd16743281d21d7fb38a34ac3" gracePeriod=30
Sep 16 18:10:02.331553 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:02.331383    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:02.574585 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:02.574501    2581 generic.go:356] "Generic (PLEG): container finished" podID="69461556-2d5d-4209-9468-df0cf9bd65ba" containerID="3bf1459626b62d05ad7c4b7ed23c63c368e196dfd16743281d21d7fb38a34ac3" exitCode=2
Sep 16 18:10:02.574585 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:02.574568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw" event={"ID":"69461556-2d5d-4209-9468-df0cf9bd65ba","Type":"ContainerDied","Data":"3bf1459626b62d05ad7c4b7ed23c63c368e196dfd16743281d21d7fb38a34ac3"}
Sep 16 18:10:03.265651 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.265628    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:10:03.388377 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.388306    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69461556-2d5d-4209-9468-df0cf9bd65ba-cabundle-cert\") pod \"69461556-2d5d-4209-9468-df0cf9bd65ba\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") "
Sep 16 18:10:03.388377 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.388355    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69461556-2d5d-4209-9468-df0cf9bd65ba-kserve-provision-location\") pod \"69461556-2d5d-4209-9468-df0cf9bd65ba\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") "
Sep 16 18:10:03.388789 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.388378    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmqp7\" (UniqueName: \"kubernetes.io/projected/69461556-2d5d-4209-9468-df0cf9bd65ba-kube-api-access-nmqp7\") pod \"69461556-2d5d-4209-9468-df0cf9bd65ba\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") "
Sep 16 18:10:03.388789 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.388455    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69461556-2d5d-4209-9468-df0cf9bd65ba-proxy-tls\") pod \"69461556-2d5d-4209-9468-df0cf9bd65ba\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") "
Sep 16 18:10:03.388789 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.388498    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69461556-2d5d-4209-9468-df0cf9bd65ba-kube-rbac-proxy-sar-config\") pod \"69461556-2d5d-4209-9468-df0cf9bd65ba\" (UID: \"69461556-2d5d-4209-9468-df0cf9bd65ba\") "
Sep 16 18:10:03.388789 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.388691    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69461556-2d5d-4209-9468-df0cf9bd65ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69461556-2d5d-4209-9468-df0cf9bd65ba" (UID: "69461556-2d5d-4209-9468-df0cf9bd65ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:10:03.388789 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.388700    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69461556-2d5d-4209-9468-df0cf9bd65ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69461556-2d5d-4209-9468-df0cf9bd65ba" (UID: "69461556-2d5d-4209-9468-df0cf9bd65ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:10:03.389085 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.388997    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69461556-2d5d-4209-9468-df0cf9bd65ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69461556-2d5d-4209-9468-df0cf9bd65ba" (UID: "69461556-2d5d-4209-9468-df0cf9bd65ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:10:03.390703 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.390681    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69461556-2d5d-4209-9468-df0cf9bd65ba-kube-api-access-nmqp7" (OuterVolumeSpecName: "kube-api-access-nmqp7") pod "69461556-2d5d-4209-9468-df0cf9bd65ba" (UID: "69461556-2d5d-4209-9468-df0cf9bd65ba"). InnerVolumeSpecName "kube-api-access-nmqp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:10:03.390783 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.390741    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69461556-2d5d-4209-9468-df0cf9bd65ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69461556-2d5d-4209-9468-df0cf9bd65ba" (UID: "69461556-2d5d-4209-9468-df0cf9bd65ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:10:03.489319 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.489293    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69461556-2d5d-4209-9468-df0cf9bd65ba-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:10:03.489319 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.489314    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmqp7\" (UniqueName: \"kubernetes.io/projected/69461556-2d5d-4209-9468-df0cf9bd65ba-kube-api-access-nmqp7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:10:03.489452 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.489323    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69461556-2d5d-4209-9468-df0cf9bd65ba-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:10:03.489452 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.489334    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69461556-2d5d-4209-9468-df0cf9bd65ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:10:03.489452 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.489344    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69461556-2d5d-4209-9468-df0cf9bd65ba-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:10:03.584024 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.583970    2581 generic.go:356] "Generic (PLEG): container finished" podID="69461556-2d5d-4209-9468-df0cf9bd65ba" containerID="cc1758df32b1925d852d7271d636f4f06c40f2c939cd0ab2413658bce134e4f2" exitCode=0
Sep 16 18:10:03.584148 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.584045    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw" event={"ID":"69461556-2d5d-4209-9468-df0cf9bd65ba","Type":"ContainerDied","Data":"cc1758df32b1925d852d7271d636f4f06c40f2c939cd0ab2413658bce134e4f2"}
Sep 16 18:10:03.584148 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.584079    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"
Sep 16 18:10:03.584148 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.584144    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw" event={"ID":"69461556-2d5d-4209-9468-df0cf9bd65ba","Type":"ContainerDied","Data":"1fea6c125f8a6c2e6dc9b1478acf628eb43eac60c0f6175d894b054cd97826cf"}
Sep 16 18:10:03.584306 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.584193    2581 scope.go:117] "RemoveContainer" containerID="3bf1459626b62d05ad7c4b7ed23c63c368e196dfd16743281d21d7fb38a34ac3"
Sep 16 18:10:03.593885 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.593864    2581 scope.go:117] "RemoveContainer" containerID="cc1758df32b1925d852d7271d636f4f06c40f2c939cd0ab2413658bce134e4f2"
Sep 16 18:10:03.601530 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.601513    2581 scope.go:117] "RemoveContainer" containerID="dfc729f62ee525d56e5e90a5aed5aed9fafd168929e54240c1e7d3311bb8d887"
Sep 16 18:10:03.608335 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.608314    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"]
Sep 16 18:10:03.609228 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.609211    2581 scope.go:117] "RemoveContainer" containerID="3bf1459626b62d05ad7c4b7ed23c63c368e196dfd16743281d21d7fb38a34ac3"
Sep 16 18:10:03.609486 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:10:03.609466    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf1459626b62d05ad7c4b7ed23c63c368e196dfd16743281d21d7fb38a34ac3\": container with ID starting with 3bf1459626b62d05ad7c4b7ed23c63c368e196dfd16743281d21d7fb38a34ac3 not found: ID does not exist" containerID="3bf1459626b62d05ad7c4b7ed23c63c368e196dfd16743281d21d7fb38a34ac3"
Sep 16 18:10:03.609533 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.609493    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf1459626b62d05ad7c4b7ed23c63c368e196dfd16743281d21d7fb38a34ac3"} err="failed to get container status \"3bf1459626b62d05ad7c4b7ed23c63c368e196dfd16743281d21d7fb38a34ac3\": rpc error: code = NotFound desc = could not find container \"3bf1459626b62d05ad7c4b7ed23c63c368e196dfd16743281d21d7fb38a34ac3\": container with ID starting with 3bf1459626b62d05ad7c4b7ed23c63c368e196dfd16743281d21d7fb38a34ac3 not found: ID does not exist"
Sep 16 18:10:03.609533 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.609510    2581 scope.go:117] "RemoveContainer" containerID="cc1758df32b1925d852d7271d636f4f06c40f2c939cd0ab2413658bce134e4f2"
Sep 16 18:10:03.609743 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:10:03.609725    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc1758df32b1925d852d7271d636f4f06c40f2c939cd0ab2413658bce134e4f2\": container with ID starting with cc1758df32b1925d852d7271d636f4f06c40f2c939cd0ab2413658bce134e4f2 not found: ID does not exist" containerID="cc1758df32b1925d852d7271d636f4f06c40f2c939cd0ab2413658bce134e4f2"
Sep 16 18:10:03.609784 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.609748    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc1758df32b1925d852d7271d636f4f06c40f2c939cd0ab2413658bce134e4f2"} err="failed to get container status \"cc1758df32b1925d852d7271d636f4f06c40f2c939cd0ab2413658bce134e4f2\": rpc error: code = NotFound desc = could not find container \"cc1758df32b1925d852d7271d636f4f06c40f2c939cd0ab2413658bce134e4f2\": container with ID starting with cc1758df32b1925d852d7271d636f4f06c40f2c939cd0ab2413658bce134e4f2 not found: ID does not exist"
Sep 16 18:10:03.609784 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.609761    2581 scope.go:117] "RemoveContainer" containerID="dfc729f62ee525d56e5e90a5aed5aed9fafd168929e54240c1e7d3311bb8d887"
Sep 16 18:10:03.610033 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:10:03.610015    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfc729f62ee525d56e5e90a5aed5aed9fafd168929e54240c1e7d3311bb8d887\": container with ID starting with dfc729f62ee525d56e5e90a5aed5aed9fafd168929e54240c1e7d3311bb8d887 not found: ID does not exist" containerID="dfc729f62ee525d56e5e90a5aed5aed9fafd168929e54240c1e7d3311bb8d887"
Sep 16 18:10:03.610107 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.610042    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfc729f62ee525d56e5e90a5aed5aed9fafd168929e54240c1e7d3311bb8d887"} err="failed to get container status \"dfc729f62ee525d56e5e90a5aed5aed9fafd168929e54240c1e7d3311bb8d887\": rpc error: code = NotFound desc = could not find container \"dfc729f62ee525d56e5e90a5aed5aed9fafd168929e54240c1e7d3311bb8d887\": container with ID starting with dfc729f62ee525d56e5e90a5aed5aed9fafd168929e54240c1e7d3311bb8d887 not found: ID does not exist"
Sep 16 18:10:03.613761 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.613742    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxpsw"]
Sep 16 18:10:03.766143 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:03.766083    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69461556-2d5d-4209-9468-df0cf9bd65ba" path="/var/lib/kubelet/pods/69461556-2d5d-4209-9468-df0cf9bd65ba/volumes"
Sep 16 18:10:07.452983 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.452952    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:07.525181 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525148    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"]
Sep 16 18:10:07.525462 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525450    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="storage-initializer"
Sep 16 18:10:07.525516 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525464    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="storage-initializer"
Sep 16 18:10:07.525516 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525477    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="kserve-container"
Sep 16 18:10:07.525516 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525482    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="kserve-container"
Sep 16 18:10:07.525516 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525490    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="kube-rbac-proxy"
Sep 16 18:10:07.525516 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525495    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="kube-rbac-proxy"
Sep 16 18:10:07.525516 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525504    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69461556-2d5d-4209-9468-df0cf9bd65ba" containerName="kube-rbac-proxy"
Sep 16 18:10:07.525516 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525509    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="69461556-2d5d-4209-9468-df0cf9bd65ba" containerName="kube-rbac-proxy"
Sep 16 18:10:07.525516 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525515    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69461556-2d5d-4209-9468-df0cf9bd65ba" containerName="kserve-container"
Sep 16 18:10:07.525764 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525521    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="69461556-2d5d-4209-9468-df0cf9bd65ba" containerName="kserve-container"
Sep 16 18:10:07.525764 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525528    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69461556-2d5d-4209-9468-df0cf9bd65ba" containerName="storage-initializer"
Sep 16 18:10:07.525764 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525533    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="69461556-2d5d-4209-9468-df0cf9bd65ba" containerName="storage-initializer"
Sep 16 18:10:07.525764 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525579    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="69461556-2d5d-4209-9468-df0cf9bd65ba" containerName="kserve-container"
Sep 16 18:10:07.525764 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525587    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="kube-rbac-proxy"
Sep 16 18:10:07.525764 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525593    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e64b1f1a-5f48-42ce-a5fc-94a9110fd75a" containerName="kserve-container"
Sep 16 18:10:07.525764 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.525601    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="69461556-2d5d-4209-9468-df0cf9bd65ba" containerName="kube-rbac-proxy"
Sep 16 18:10:07.530350 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.530319    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.533879 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.533856    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:10:07.534013 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.533879    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\""
Sep 16 18:10:07.534013 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.533857    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:10:07.534013 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.533926    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:10:07.534918 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.534874    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:10:07.535016 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.534874    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 18:10:07.535016 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.534982    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 16 18:10:07.541301 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.541278    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"]
Sep 16 18:10:07.579423 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.579391    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:07.622601 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.622572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.622721 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.622607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.622721 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.622626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.622799 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.622728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9w7z\" (UniqueName: \"kubernetes.io/projected/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kube-api-access-x9w7z\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.622799 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.622773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.723776 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.723697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.723776 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.723734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.724021 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.723752    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.724021 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.723833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.724021 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.723924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9w7z\" (UniqueName: \"kubernetes.io/projected/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kube-api-access-x9w7z\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.724288 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.724269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.724541 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.724519    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.724590 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.724519    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.726445 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.726422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.733916 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.733881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9w7z\" (UniqueName: \"kubernetes.io/projected/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kube-api-access-x9w7z\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:07.840798 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:07.840766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:08.123415 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:08.123236    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"]
Sep 16 18:10:08.126097 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:10:08.126065    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95eaf0a1_ec4a_48d7_8957_aecaa4f7f131.slice/crio-b3aa50a99f27e7015588e295f755998a6b2629c7d3d13d953c36a4d5beebb358 WatchSource:0}: Error finding container b3aa50a99f27e7015588e295f755998a6b2629c7d3d13d953c36a4d5beebb358: Status 404 returned error can't find the container with id b3aa50a99f27e7015588e295f755998a6b2629c7d3d13d953c36a4d5beebb358
Sep 16 18:10:08.610122 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:08.610079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5" event={"ID":"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131","Type":"ContainerStarted","Data":"f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f"}
Sep 16 18:10:08.610618 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:08.610153    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5" event={"ID":"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131","Type":"ContainerStarted","Data":"b3aa50a99f27e7015588e295f755998a6b2629c7d3d13d953c36a4d5beebb358"}
Sep 16 18:10:12.628585 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:12.628555    2581 generic.go:356] "Generic (PLEG): container finished" podID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" containerID="f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f" exitCode=0
Sep 16 18:10:12.629042 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:12.628634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5" event={"ID":"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131","Type":"ContainerDied","Data":"f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f"}
Sep 16 18:10:19.666002 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:19.665962    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5" event={"ID":"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131","Type":"ContainerStarted","Data":"11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d"}
Sep 16 18:10:19.666465 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:19.666033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5" event={"ID":"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131","Type":"ContainerStarted","Data":"82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7"}
Sep 16 18:10:19.666465 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:19.666232    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:19.701152 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:19.701090    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5" podStartSLOduration=2.8569011250000003 podStartE2EDuration="12.701071138s" totalImagesPullingTime="6.221852206s" totalInitContainerRuntime="3.622317807s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:10:19.698026832 +0000 UTC m=+2748.486649365" watchObservedRunningTime="2026-09-16 18:10:19.701071138 +0000 UTC m=+2748.489693673"
Sep 16 18:10:20.670437 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:20.670404    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:20.671743 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:20.671713    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5" podUID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 16 18:10:21.675075 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:21.675028    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5" podUID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 16 18:10:26.679651 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:26.679622    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:26.680658 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:26.680641    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:27.635743 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:27.635713    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"]
Sep 16 18:10:27.668701 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:27.668662    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:27.699938 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:27.699903    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5" podUID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" containerName="kserve-container" containerID="cri-o://82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7" gracePeriod=30
Sep 16 18:10:27.700304 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:27.699938    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5" podUID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" containerName="kube-rbac-proxy" containerID="cri-o://11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d" gracePeriod=30
Sep 16 18:10:28.348493 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.348469    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:28.497029 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.496923    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9w7z\" (UniqueName: \"kubernetes.io/projected/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kube-api-access-x9w7z\") pod \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") "
Sep 16 18:10:28.497029 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.496988    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-cabundle-cert\") pod \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") "
Sep 16 18:10:28.497268 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.497068    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kube-rbac-proxy-sar-config\") pod \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") "
Sep 16 18:10:28.497268 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.497118    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kserve-provision-location\") pod \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") "
Sep 16 18:10:28.497268 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.497171    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-proxy-tls\") pod \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\" (UID: \"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131\") "
Sep 16 18:10:28.497419 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.497348    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" (UID: "95eaf0a1-ec4a-48d7-8957-aecaa4f7f131"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:10:28.497476 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.497458    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" (UID: "95eaf0a1-ec4a-48d7-8957-aecaa4f7f131"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:10:28.497580 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.497552    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" (UID: "95eaf0a1-ec4a-48d7-8957-aecaa4f7f131"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:10:28.499356 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.499334    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kube-api-access-x9w7z" (OuterVolumeSpecName: "kube-api-access-x9w7z") pod "95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" (UID: "95eaf0a1-ec4a-48d7-8957-aecaa4f7f131"). InnerVolumeSpecName "kube-api-access-x9w7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:10:28.499356 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.499334    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" (UID: "95eaf0a1-ec4a-48d7-8957-aecaa4f7f131"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:10:28.598111 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.598082    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:10:28.598111 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.598106    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:10:28.598111 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.598117    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9w7z\" (UniqueName: \"kubernetes.io/projected/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kube-api-access-x9w7z\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:10:28.598312 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.598128    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:10:28.598312 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.598137    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:10:28.706240 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.706206    2581 generic.go:356] "Generic (PLEG): container finished" podID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" containerID="11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d" exitCode=2
Sep 16 18:10:28.706240 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.706233    2581 generic.go:356] "Generic (PLEG): container finished" podID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" containerID="82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7" exitCode=0
Sep 16 18:10:28.706670 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.706283    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"
Sep 16 18:10:28.706670 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.706286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5" event={"ID":"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131","Type":"ContainerDied","Data":"11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d"}
Sep 16 18:10:28.706670 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.706346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5" event={"ID":"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131","Type":"ContainerDied","Data":"82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7"}
Sep 16 18:10:28.706670 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.706368    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5" event={"ID":"95eaf0a1-ec4a-48d7-8957-aecaa4f7f131","Type":"ContainerDied","Data":"b3aa50a99f27e7015588e295f755998a6b2629c7d3d13d953c36a4d5beebb358"}
Sep 16 18:10:28.706670 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.706392    2581 scope.go:117] "RemoveContainer" containerID="11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d"
Sep 16 18:10:28.717376 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.717356    2581 scope.go:117] "RemoveContainer" containerID="82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7"
Sep 16 18:10:28.724769 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.724741    2581 scope.go:117] "RemoveContainer" containerID="f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f"
Sep 16 18:10:28.730113 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.730094    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"]
Sep 16 18:10:28.733444 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.732576    2581 scope.go:117] "RemoveContainer" containerID="11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d"
Sep 16 18:10:28.733444 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:10:28.732911    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d\": container with ID starting with 11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d not found: ID does not exist" containerID="11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d"
Sep 16 18:10:28.733444 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.732945    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d"} err="failed to get container status \"11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d\": rpc error: code = NotFound desc = could not find container \"11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d\": container with ID starting with 11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d not found: ID does not exist"
Sep 16 18:10:28.733444 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.732968    2581 scope.go:117] "RemoveContainer" containerID="82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7"
Sep 16 18:10:28.733444 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:10:28.733237    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7\": container with ID starting with 82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7 not found: ID does not exist" containerID="82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7"
Sep 16 18:10:28.733444 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.733263    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7"} err="failed to get container status \"82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7\": rpc error: code = NotFound desc = could not find container \"82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7\": container with ID starting with 82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7 not found: ID does not exist"
Sep 16 18:10:28.733444 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.733282    2581 scope.go:117] "RemoveContainer" containerID="f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f"
Sep 16 18:10:28.733880 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:10:28.733621    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f\": container with ID starting with f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f not found: ID does not exist" containerID="f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f"
Sep 16 18:10:28.733880 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.733653    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f"} err="failed to get container status \"f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f\": rpc error: code = NotFound desc = could not find container \"f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f\": container with ID starting with f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f not found: ID does not exist"
Sep 16 18:10:28.733880 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.733676    2581 scope.go:117] "RemoveContainer" containerID="11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d"
Sep 16 18:10:28.734110 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.734016    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d"} err="failed to get container status \"11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d\": rpc error: code = NotFound desc = could not find container \"11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d\": container with ID starting with 11b1a956a0b5356c5b9cbe7c36f61ca3b3dde4e60b0b92ad25bfd1c6b5d0138d not found: ID does not exist"
Sep 16 18:10:28.734110 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.734039    2581 scope.go:117] "RemoveContainer" containerID="82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7"
Sep 16 18:10:28.734312 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.734294    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7"} err="failed to get container status \"82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7\": rpc error: code = NotFound desc = could not find container \"82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7\": container with ID starting with 82155d7a8772fc86b6567bc21e08165c735984e555d2347548f9064ce67862f7 not found: ID does not exist"
Sep 16 18:10:28.734357 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.734312    2581 scope.go:117] "RemoveContainer" containerID="f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f"
Sep 16 18:10:28.734572 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.734553    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f"} err="failed to get container status \"f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f\": rpc error: code = NotFound desc = could not find container \"f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f\": container with ID starting with f69358d4df88835d5cde2a225e6e64fdaddb901ff39fe87bda4600c483cb150f not found: ID does not exist"
Sep 16 18:10:28.734816 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:28.734798    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xmdl5"]
Sep 16 18:10:29.766279 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:29.766245    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" path="/var/lib/kubelet/pods/95eaf0a1-ec4a-48d7-8957-aecaa4f7f131/volumes"
Sep 16 18:10:32.743932 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.743876    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:32.783121 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.783094    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"]
Sep 16 18:10:32.783416 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.783404    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" containerName="kube-rbac-proxy"
Sep 16 18:10:32.783459 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.783418    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" containerName="kube-rbac-proxy"
Sep 16 18:10:32.783459 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.783430    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" containerName="storage-initializer"
Sep 16 18:10:32.783459 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.783435    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" containerName="storage-initializer"
Sep 16 18:10:32.783459 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.783444    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" containerName="kserve-container"
Sep 16 18:10:32.783459 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.783449    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" containerName="kserve-container"
Sep 16 18:10:32.783616 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.783509    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" containerName="kserve-container"
Sep 16 18:10:32.783616 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.783517    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95eaf0a1-ec4a-48d7-8957-aecaa4f7f131" containerName="kube-rbac-proxy"
Sep 16 18:10:32.788186 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.788159    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:32.794721 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.794325    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:10:32.794721 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.794416    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:10:32.794721 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.794641    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:10:32.795071 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.795001    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\""
Sep 16 18:10:32.795071 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.795060    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 16 18:10:32.795239 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.795065    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:10:32.795239 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.795103    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 18:10:32.812872 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.812847    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"]
Sep 16 18:10:32.901924 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.901878    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:32.930482 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.930462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttjtq\" (UniqueName: \"kubernetes.io/projected/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kube-api-access-ttjtq\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:32.930571 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.930494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:32.930571 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.930515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:32.930650 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.930572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:32.930650 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:32.930622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:33.031762 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:33.031741    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:33.031855 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:33.031784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:33.031855 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:33.031824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttjtq\" (UniqueName: \"kubernetes.io/projected/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kube-api-access-ttjtq\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:33.031855 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:33.031846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:33.032002 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:33.031869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:33.032298 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:33.032276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:33.032415 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:33.032387    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:33.032550 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:33.032531    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:33.034561 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:33.034545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:33.043265 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:33.043244    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttjtq\" (UniqueName: \"kubernetes.io/projected/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kube-api-access-ttjtq\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:33.098330 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:33.098304    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:33.380773 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:33.380696    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"]
Sep 16 18:10:33.383782 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:10:33.383747    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56942ff4_8777_42d3_ab5d_d2a216b3e2cc.slice/crio-4825a10d957ae2137c23a30e52a4c91facc4fcd1e60cac9051f47b84b7e29ceb WatchSource:0}: Error finding container 4825a10d957ae2137c23a30e52a4c91facc4fcd1e60cac9051f47b84b7e29ceb: Status 404 returned error can't find the container with id 4825a10d957ae2137c23a30e52a4c91facc4fcd1e60cac9051f47b84b7e29ceb
Sep 16 18:10:33.731866 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:33.731776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2" event={"ID":"56942ff4-8777-42d3-ab5d-d2a216b3e2cc","Type":"ContainerStarted","Data":"43f083e4aca2ddecdc7a65bac0c927178eedea5f4a6087c3b3d61f461125313a"}
Sep 16 18:10:33.731866 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:33.731835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2" event={"ID":"56942ff4-8777-42d3-ab5d-d2a216b3e2cc","Type":"ContainerStarted","Data":"4825a10d957ae2137c23a30e52a4c91facc4fcd1e60cac9051f47b84b7e29ceb"}
Sep 16 18:10:37.751875 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:37.751795    2581 generic.go:356] "Generic (PLEG): container finished" podID="56942ff4-8777-42d3-ab5d-d2a216b3e2cc" containerID="43f083e4aca2ddecdc7a65bac0c927178eedea5f4a6087c3b3d61f461125313a" exitCode=0
Sep 16 18:10:37.752267 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:37.751873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2" event={"ID":"56942ff4-8777-42d3-ab5d-d2a216b3e2cc","Type":"ContainerDied","Data":"43f083e4aca2ddecdc7a65bac0c927178eedea5f4a6087c3b3d61f461125313a"}
Sep 16 18:10:38.766648 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:38.766609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2" event={"ID":"56942ff4-8777-42d3-ab5d-d2a216b3e2cc","Type":"ContainerStarted","Data":"dd2e765e19fa09a9696359f60924b4b34e3cdd85aecbf081f4bb226de29f2123"}
Sep 16 18:10:38.767242 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:38.766708    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2" event={"ID":"56942ff4-8777-42d3-ab5d-d2a216b3e2cc","Type":"ContainerStarted","Data":"26046f889b3c1df82daa4649c07ec1c5eb607e372782068009d5a91895efb019"}
Sep 16 18:10:38.767242 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:38.767107    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:38.818223 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:38.818182    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2" podStartSLOduration=2.9306048049999998 podStartE2EDuration="6.81816524s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.887560435s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:10:38.817761287 +0000 UTC m=+2767.606383821" watchObservedRunningTime="2026-09-16 18:10:38.81816524 +0000 UTC m=+2767.606787774"
Sep 16 18:10:39.770757 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:39.770727    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:45.779759 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:45.779734    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:10:50.067583 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:50.067551    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:50.263162 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:50.263133    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:50.609274 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:10:50.609235    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:15.784149 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:15.784055    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:11:22.828128 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:22.828095    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"]
Sep 16 18:11:22.828591 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:22.828399    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2" podUID="56942ff4-8777-42d3-ab5d-d2a216b3e2cc" containerName="kserve-container" containerID="cri-o://26046f889b3c1df82daa4649c07ec1c5eb607e372782068009d5a91895efb019" gracePeriod=30
Sep 16 18:11:22.828591 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:22.828437    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2" podUID="56942ff4-8777-42d3-ab5d-d2a216b3e2cc" containerName="kube-rbac-proxy" containerID="cri-o://dd2e765e19fa09a9696359f60924b4b34e3cdd85aecbf081f4bb226de29f2123" gracePeriod=30
Sep 16 18:11:22.839768 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:22.839717    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:22.950201 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:22.950172    2581 generic.go:356] "Generic (PLEG): container finished" podID="56942ff4-8777-42d3-ab5d-d2a216b3e2cc" containerID="dd2e765e19fa09a9696359f60924b4b34e3cdd85aecbf081f4bb226de29f2123" exitCode=2
Sep 16 18:11:22.950344 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:22.950241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2" event={"ID":"56942ff4-8777-42d3-ab5d-d2a216b3e2cc","Type":"ContainerDied","Data":"dd2e765e19fa09a9696359f60924b4b34e3cdd85aecbf081f4bb226de29f2123"}
Sep 16 18:11:23.956711 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:23.956628    2581 generic.go:356] "Generic (PLEG): container finished" podID="56942ff4-8777-42d3-ab5d-d2a216b3e2cc" containerID="26046f889b3c1df82daa4649c07ec1c5eb607e372782068009d5a91895efb019" exitCode=0
Sep 16 18:11:23.957111 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:23.956705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2" event={"ID":"56942ff4-8777-42d3-ab5d-d2a216b3e2cc","Type":"ContainerDied","Data":"26046f889b3c1df82daa4649c07ec1c5eb607e372782068009d5a91895efb019"}
Sep 16 18:11:23.970225 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:23.970206    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:11:24.103253 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.103226    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kserve-provision-location\") pod \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") "
Sep 16 18:11:24.103371 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.103301    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-proxy-tls\") pod \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") "
Sep 16 18:11:24.103371 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.103343    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttjtq\" (UniqueName: \"kubernetes.io/projected/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kube-api-access-ttjtq\") pod \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") "
Sep 16 18:11:24.103455 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.103371    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kube-rbac-proxy-sar-config\") pod \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") "
Sep 16 18:11:24.103455 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.103429    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-cabundle-cert\") pod \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\" (UID: \"56942ff4-8777-42d3-ab5d-d2a216b3e2cc\") "
Sep 16 18:11:24.103614 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.103592    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "56942ff4-8777-42d3-ab5d-d2a216b3e2cc" (UID: "56942ff4-8777-42d3-ab5d-d2a216b3e2cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:11:24.103709 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.103691    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:11:24.103843 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.103819    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "56942ff4-8777-42d3-ab5d-d2a216b3e2cc" (UID: "56942ff4-8777-42d3-ab5d-d2a216b3e2cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:11:24.103942 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.103885    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "56942ff4-8777-42d3-ab5d-d2a216b3e2cc" (UID: "56942ff4-8777-42d3-ab5d-d2a216b3e2cc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:11:24.105484 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.105466    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56942ff4-8777-42d3-ab5d-d2a216b3e2cc" (UID: "56942ff4-8777-42d3-ab5d-d2a216b3e2cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:11:24.105574 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.105564    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kube-api-access-ttjtq" (OuterVolumeSpecName: "kube-api-access-ttjtq") pod "56942ff4-8777-42d3-ab5d-d2a216b3e2cc" (UID: "56942ff4-8777-42d3-ab5d-d2a216b3e2cc"). InnerVolumeSpecName "kube-api-access-ttjtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:11:24.204356 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.204329    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:11:24.204356 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.204352    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttjtq\" (UniqueName: \"kubernetes.io/projected/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kube-api-access-ttjtq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:11:24.204494 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.204362    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:11:24.204494 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.204371    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56942ff4-8777-42d3-ab5d-d2a216b3e2cc-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:11:24.964710 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.964688    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"
Sep 16 18:11:24.965148 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.964684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2" event={"ID":"56942ff4-8777-42d3-ab5d-d2a216b3e2cc","Type":"ContainerDied","Data":"4825a10d957ae2137c23a30e52a4c91facc4fcd1e60cac9051f47b84b7e29ceb"}
Sep 16 18:11:24.965148 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.964860    2581 scope.go:117] "RemoveContainer" containerID="dd2e765e19fa09a9696359f60924b4b34e3cdd85aecbf081f4bb226de29f2123"
Sep 16 18:11:24.974716 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.974694    2581 scope.go:117] "RemoveContainer" containerID="26046f889b3c1df82daa4649c07ec1c5eb607e372782068009d5a91895efb019"
Sep 16 18:11:24.982830 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.982810    2581 scope.go:117] "RemoveContainer" containerID="43f083e4aca2ddecdc7a65bac0c927178eedea5f4a6087c3b3d61f461125313a"
Sep 16 18:11:24.991611 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.991590    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"]
Sep 16 18:11:24.996732 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:24.996708    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-q5qq2"]
Sep 16 18:11:25.766980 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:25.766946    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56942ff4-8777-42d3-ab5d-d2a216b3e2cc" path="/var/lib/kubelet/pods/56942ff4-8777-42d3-ab5d-d2a216b3e2cc/volumes"
Sep 16 18:11:28.002056 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.002028    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:28.078350 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.078317    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"]
Sep 16 18:11:28.078648 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.078634    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56942ff4-8777-42d3-ab5d-d2a216b3e2cc" containerName="kserve-container"
Sep 16 18:11:28.078694 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.078654    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="56942ff4-8777-42d3-ab5d-d2a216b3e2cc" containerName="kserve-container"
Sep 16 18:11:28.078694 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.078664    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56942ff4-8777-42d3-ab5d-d2a216b3e2cc" containerName="kube-rbac-proxy"
Sep 16 18:11:28.078694 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.078669    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="56942ff4-8777-42d3-ab5d-d2a216b3e2cc" containerName="kube-rbac-proxy"
Sep 16 18:11:28.078694 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.078677    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56942ff4-8777-42d3-ab5d-d2a216b3e2cc" containerName="storage-initializer"
Sep 16 18:11:28.078694 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.078683    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="56942ff4-8777-42d3-ab5d-d2a216b3e2cc" containerName="storage-initializer"
Sep 16 18:11:28.078853 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.078737    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="56942ff4-8777-42d3-ab5d-d2a216b3e2cc" containerName="kube-rbac-proxy"
Sep 16 18:11:28.078853 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.078750    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="56942ff4-8777-42d3-ab5d-d2a216b3e2cc" containerName="kserve-container"
Sep 16 18:11:28.083443 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.083427    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.090061 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.090036    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:11:28.090180 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.090159    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:11:28.090254 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.090183    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:11:28.090254 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.090102    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 16 18:11:28.090254 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.090075    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 16 18:11:28.090408 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.090310    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\""
Sep 16 18:11:28.091072 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.091042    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:11:28.098851 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.098830    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"]
Sep 16 18:11:28.133793 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.133774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/993ca158-d110-45c4-a863-113939c64342-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.133915 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.133810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/993ca158-d110-45c4-a863-113939c64342-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.133915 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.133841    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/993ca158-d110-45c4-a863-113939c64342-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.133915 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.133868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/993ca158-d110-45c4-a863-113939c64342-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.133915 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.133883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sxg6\" (UniqueName: \"kubernetes.io/projected/993ca158-d110-45c4-a863-113939c64342-kube-api-access-2sxg6\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.235097 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.235079    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/993ca158-d110-45c4-a863-113939c64342-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.235207 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.235111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/993ca158-d110-45c4-a863-113939c64342-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.235207 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.235133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/993ca158-d110-45c4-a863-113939c64342-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.235317 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:11:28.235211    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 16 18:11:28.235317 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:11:28.235261    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/993ca158-d110-45c4-a863-113939c64342-proxy-tls podName:993ca158-d110-45c4-a863-113939c64342 nodeName:}" failed. No retries permitted until 2026-09-16 18:11:28.73524627 +0000 UTC m=+2817.523868796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/993ca158-d110-45c4-a863-113939c64342-proxy-tls") pod "isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" (UID: "993ca158-d110-45c4-a863-113939c64342") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 16 18:11:28.235317 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.235278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/993ca158-d110-45c4-a863-113939c64342-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.235317 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.235297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sxg6\" (UniqueName: \"kubernetes.io/projected/993ca158-d110-45c4-a863-113939c64342-kube-api-access-2sxg6\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.235561 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.235539    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/993ca158-d110-45c4-a863-113939c64342-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.235830 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.235809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/993ca158-d110-45c4-a863-113939c64342-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.235883 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.235864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/993ca158-d110-45c4-a863-113939c64342-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.252154 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.252106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sxg6\" (UniqueName: \"kubernetes.io/projected/993ca158-d110-45c4-a863-113939c64342-kube-api-access-2sxg6\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.389440 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.389411    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:28.738509 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.738469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/993ca158-d110-45c4-a863-113939c64342-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.741363 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.741328    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/993ca158-d110-45c4-a863-113939c64342-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:28.995785 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:28.995712    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:29.284012 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:29.283918    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"]
Sep 16 18:11:29.287070 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:11:29.287032    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993ca158_d110_45c4_a863_113939c64342.slice/crio-039fff764dcb79494c2f3c3283dc9132c77767cc2cebd3b6bc69447133449635 WatchSource:0}: Error finding container 039fff764dcb79494c2f3c3283dc9132c77767cc2cebd3b6bc69447133449635: Status 404 returned error can't find the container with id 039fff764dcb79494c2f3c3283dc9132c77767cc2cebd3b6bc69447133449635
Sep 16 18:11:29.989777 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:29.989744    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" event={"ID":"993ca158-d110-45c4-a863-113939c64342","Type":"ContainerStarted","Data":"e2fb05b60a90e04f59f97090dfdd49608d6bdffa0683e1c55eb0918f1571b01f"}
Sep 16 18:11:29.990044 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:29.989825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" event={"ID":"993ca158-d110-45c4-a863-113939c64342","Type":"ContainerStarted","Data":"039fff764dcb79494c2f3c3283dc9132c77767cc2cebd3b6bc69447133449635"}
Sep 16 18:11:34.008419 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:34.008386    2581 generic.go:356] "Generic (PLEG): container finished" podID="993ca158-d110-45c4-a863-113939c64342" containerID="e2fb05b60a90e04f59f97090dfdd49608d6bdffa0683e1c55eb0918f1571b01f" exitCode=0
Sep 16 18:11:34.008778 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:34.008457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" event={"ID":"993ca158-d110-45c4-a863-113939c64342","Type":"ContainerDied","Data":"e2fb05b60a90e04f59f97090dfdd49608d6bdffa0683e1c55eb0918f1571b01f"}
Sep 16 18:11:34.075985 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:34.075957    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:11:35.016542 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:35.016509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" event={"ID":"993ca158-d110-45c4-a863-113939c64342","Type":"ContainerStarted","Data":"739f604f49ff24371c3b132500d6e77915cfded0f01df2b84b10fa8210ad3799"}
Sep 16 18:11:37.027613 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:37.027536    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" event={"ID":"993ca158-d110-45c4-a863-113939c64342","Type":"ContainerStarted","Data":"327cfb4e17f6a561f71ce299951b5ce40a77b429f9a35bf836700cc30573878d"}
Sep 16 18:11:37.027973 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:37.027627    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:37.027973 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:37.027651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" event={"ID":"993ca158-d110-45c4-a863-113939c64342","Type":"ContainerStarted","Data":"974c944c66f64c4dadd6bd085d7368833b7e72bc532172dfd4e4a38db9ae2138"}
Sep 16 18:11:37.027973 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:37.027759    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:37.060583 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:37.060528    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" podStartSLOduration=2.7034953489999998 podStartE2EDuration="9.060511725s" totalImagesPullingTime="2.696473348s" totalInitContainerRuntime="3.660543028s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:11:37.059559878 +0000 UTC m=+2825.848182411" watchObservedRunningTime="2026-09-16 18:11:37.060511725 +0000 UTC m=+2825.849134267"
Sep 16 18:11:38.031474 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:38.031448    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:11:44.041008 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:11:44.040977    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:12:04.042084 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:04.042044    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.62:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 16 18:12:10.247914 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:10.247859    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:12:11.322921 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:11.322861    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:12:11.434229 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:11.434196    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:12:14.043355 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:14.043326    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:12:21.336272 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:21.336237    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:12:22.450627 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:22.450583    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:12:22.701884 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:22.701771    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:12:40.189483 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:40.189446    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:12:44.044477 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:44.044450    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:12:48.156507 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:48.156469    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"]
Sep 16 18:12:48.157033 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:48.156722    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kserve-container" containerID="cri-o://739f604f49ff24371c3b132500d6e77915cfded0f01df2b84b10fa8210ad3799" gracePeriod=30
Sep 16 18:12:48.157033 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:48.156764    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kube-rbac-proxy" containerID="cri-o://327cfb4e17f6a561f71ce299951b5ce40a77b429f9a35bf836700cc30573878d" gracePeriod=30
Sep 16 18:12:48.157033 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:48.156788    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kserve-agent" containerID="cri-o://974c944c66f64c4dadd6bd085d7368833b7e72bc532172dfd4e4a38db9ae2138" gracePeriod=30
Sep 16 18:12:48.160010 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:48.159986    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:12:48.321694 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:48.321664    2581 generic.go:356] "Generic (PLEG): container finished" podID="993ca158-d110-45c4-a863-113939c64342" containerID="327cfb4e17f6a561f71ce299951b5ce40a77b429f9a35bf836700cc30573878d" exitCode=2
Sep 16 18:12:48.321866 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:48.321722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" event={"ID":"993ca158-d110-45c4-a863-113939c64342","Type":"ContainerDied","Data":"327cfb4e17f6a561f71ce299951b5ce40a77b429f9a35bf836700cc30573878d"}
Sep 16 18:12:49.036252 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:49.036205    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Sep 16 18:12:51.337016 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:51.336985    2581 generic.go:356] "Generic (PLEG): container finished" podID="993ca158-d110-45c4-a863-113939c64342" containerID="739f604f49ff24371c3b132500d6e77915cfded0f01df2b84b10fa8210ad3799" exitCode=0
Sep 16 18:12:51.337394 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:51.337038    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" event={"ID":"993ca158-d110-45c4-a863-113939c64342","Type":"ContainerDied","Data":"739f604f49ff24371c3b132500d6e77915cfded0f01df2b84b10fa8210ad3799"}
Sep 16 18:12:53.247253 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:53.247196    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:12:53.494683 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:53.494642    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:12:54.036653 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:54.036607    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Sep 16 18:12:54.042182 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:54.042146    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.62:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 16 18:12:59.036020 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:59.035983    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Sep 16 18:12:59.036461 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:12:59.036089    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:13:04.036355 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:04.036317    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Sep 16 18:13:04.041686 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:04.041663    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.62:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 16 18:13:09.036138 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:09.036087    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Sep 16 18:13:14.036718 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:14.036673    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Sep 16 18:13:14.042121 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:14.042093    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.62:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 16 18:13:14.042253 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:14.042220    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:13:18.298269 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.298245    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:13:18.454008 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.453924    2581 generic.go:356] "Generic (PLEG): container finished" podID="993ca158-d110-45c4-a863-113939c64342" containerID="974c944c66f64c4dadd6bd085d7368833b7e72bc532172dfd4e4a38db9ae2138" exitCode=0
Sep 16 18:13:18.454008 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.453986    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"
Sep 16 18:13:18.454178 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.454012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" event={"ID":"993ca158-d110-45c4-a863-113939c64342","Type":"ContainerDied","Data":"974c944c66f64c4dadd6bd085d7368833b7e72bc532172dfd4e4a38db9ae2138"}
Sep 16 18:13:18.454178 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.454081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5" event={"ID":"993ca158-d110-45c4-a863-113939c64342","Type":"ContainerDied","Data":"039fff764dcb79494c2f3c3283dc9132c77767cc2cebd3b6bc69447133449635"}
Sep 16 18:13:18.454178 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.454116    2581 scope.go:117] "RemoveContainer" containerID="327cfb4e17f6a561f71ce299951b5ce40a77b429f9a35bf836700cc30573878d"
Sep 16 18:13:18.462290 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.462273    2581 scope.go:117] "RemoveContainer" containerID="974c944c66f64c4dadd6bd085d7368833b7e72bc532172dfd4e4a38db9ae2138"
Sep 16 18:13:18.469367 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.469348    2581 scope.go:117] "RemoveContainer" containerID="739f604f49ff24371c3b132500d6e77915cfded0f01df2b84b10fa8210ad3799"
Sep 16 18:13:18.476525 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.476507    2581 scope.go:117] "RemoveContainer" containerID="e2fb05b60a90e04f59f97090dfdd49608d6bdffa0683e1c55eb0918f1571b01f"
Sep 16 18:13:18.483617 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.483599    2581 scope.go:117] "RemoveContainer" containerID="327cfb4e17f6a561f71ce299951b5ce40a77b429f9a35bf836700cc30573878d"
Sep 16 18:13:18.483714 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.483696    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/993ca158-d110-45c4-a863-113939c64342-proxy-tls\") pod \"993ca158-d110-45c4-a863-113939c64342\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") "
Sep 16 18:13:18.483773 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.483742    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/993ca158-d110-45c4-a863-113939c64342-kserve-provision-location\") pod \"993ca158-d110-45c4-a863-113939c64342\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") "
Sep 16 18:13:18.483853 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.483838    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/993ca158-d110-45c4-a863-113939c64342-kube-rbac-proxy-sar-config\") pod \"993ca158-d110-45c4-a863-113939c64342\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") "
Sep 16 18:13:18.484042 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.483885    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/993ca158-d110-45c4-a863-113939c64342-cabundle-cert\") pod \"993ca158-d110-45c4-a863-113939c64342\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") "
Sep 16 18:13:18.484042 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:13:18.483904    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"327cfb4e17f6a561f71ce299951b5ce40a77b429f9a35bf836700cc30573878d\": container with ID starting with 327cfb4e17f6a561f71ce299951b5ce40a77b429f9a35bf836700cc30573878d not found: ID does not exist" containerID="327cfb4e17f6a561f71ce299951b5ce40a77b429f9a35bf836700cc30573878d"
Sep 16 18:13:18.484042 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.483936    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"327cfb4e17f6a561f71ce299951b5ce40a77b429f9a35bf836700cc30573878d"} err="failed to get container status \"327cfb4e17f6a561f71ce299951b5ce40a77b429f9a35bf836700cc30573878d\": rpc error: code = NotFound desc = could not find container \"327cfb4e17f6a561f71ce299951b5ce40a77b429f9a35bf836700cc30573878d\": container with ID starting with 327cfb4e17f6a561f71ce299951b5ce40a77b429f9a35bf836700cc30573878d not found: ID does not exist"
Sep 16 18:13:18.484042 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.483958    2581 scope.go:117] "RemoveContainer" containerID="974c944c66f64c4dadd6bd085d7368833b7e72bc532172dfd4e4a38db9ae2138"
Sep 16 18:13:18.484042 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.483946    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sxg6\" (UniqueName: \"kubernetes.io/projected/993ca158-d110-45c4-a863-113939c64342-kube-api-access-2sxg6\") pod \"993ca158-d110-45c4-a863-113939c64342\" (UID: \"993ca158-d110-45c4-a863-113939c64342\") "
Sep 16 18:13:18.484042 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.484033    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993ca158-d110-45c4-a863-113939c64342-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "993ca158-d110-45c4-a863-113939c64342" (UID: "993ca158-d110-45c4-a863-113939c64342"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:13:18.484349 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.484220    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993ca158-d110-45c4-a863-113939c64342-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "993ca158-d110-45c4-a863-113939c64342" (UID: "993ca158-d110-45c4-a863-113939c64342"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:13:18.484349 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.484216    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993ca158-d110-45c4-a863-113939c64342-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "993ca158-d110-45c4-a863-113939c64342" (UID: "993ca158-d110-45c4-a863-113939c64342"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:13:18.484349 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.484245    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/993ca158-d110-45c4-a863-113939c64342-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:13:18.484486 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:13:18.484472    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"974c944c66f64c4dadd6bd085d7368833b7e72bc532172dfd4e4a38db9ae2138\": container with ID starting with 974c944c66f64c4dadd6bd085d7368833b7e72bc532172dfd4e4a38db9ae2138 not found: ID does not exist" containerID="974c944c66f64c4dadd6bd085d7368833b7e72bc532172dfd4e4a38db9ae2138"
Sep 16 18:13:18.484526 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.484494    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974c944c66f64c4dadd6bd085d7368833b7e72bc532172dfd4e4a38db9ae2138"} err="failed to get container status \"974c944c66f64c4dadd6bd085d7368833b7e72bc532172dfd4e4a38db9ae2138\": rpc error: code = NotFound desc = could not find container \"974c944c66f64c4dadd6bd085d7368833b7e72bc532172dfd4e4a38db9ae2138\": container with ID starting with 974c944c66f64c4dadd6bd085d7368833b7e72bc532172dfd4e4a38db9ae2138 not found: ID does not exist"
Sep 16 18:13:18.484526 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.484510    2581 scope.go:117] "RemoveContainer" containerID="739f604f49ff24371c3b132500d6e77915cfded0f01df2b84b10fa8210ad3799"
Sep 16 18:13:18.484781 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:13:18.484763    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739f604f49ff24371c3b132500d6e77915cfded0f01df2b84b10fa8210ad3799\": container with ID starting with 739f604f49ff24371c3b132500d6e77915cfded0f01df2b84b10fa8210ad3799 not found: ID does not exist" containerID="739f604f49ff24371c3b132500d6e77915cfded0f01df2b84b10fa8210ad3799"
Sep 16 18:13:18.484844 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.484785    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739f604f49ff24371c3b132500d6e77915cfded0f01df2b84b10fa8210ad3799"} err="failed to get container status \"739f604f49ff24371c3b132500d6e77915cfded0f01df2b84b10fa8210ad3799\": rpc error: code = NotFound desc = could not find container \"739f604f49ff24371c3b132500d6e77915cfded0f01df2b84b10fa8210ad3799\": container with ID starting with 739f604f49ff24371c3b132500d6e77915cfded0f01df2b84b10fa8210ad3799 not found: ID does not exist"
Sep 16 18:13:18.484844 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.484797    2581 scope.go:117] "RemoveContainer" containerID="e2fb05b60a90e04f59f97090dfdd49608d6bdffa0683e1c55eb0918f1571b01f"
Sep 16 18:13:18.485088 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:13:18.485070    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2fb05b60a90e04f59f97090dfdd49608d6bdffa0683e1c55eb0918f1571b01f\": container with ID starting with e2fb05b60a90e04f59f97090dfdd49608d6bdffa0683e1c55eb0918f1571b01f not found: ID does not exist" containerID="e2fb05b60a90e04f59f97090dfdd49608d6bdffa0683e1c55eb0918f1571b01f"
Sep 16 18:13:18.485147 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.485093    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2fb05b60a90e04f59f97090dfdd49608d6bdffa0683e1c55eb0918f1571b01f"} err="failed to get container status \"e2fb05b60a90e04f59f97090dfdd49608d6bdffa0683e1c55eb0918f1571b01f\": rpc error: code = NotFound desc = could not find container \"e2fb05b60a90e04f59f97090dfdd49608d6bdffa0683e1c55eb0918f1571b01f\": container with ID starting with e2fb05b60a90e04f59f97090dfdd49608d6bdffa0683e1c55eb0918f1571b01f not found: ID does not exist"
Sep 16 18:13:18.485975 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.485957    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/993ca158-d110-45c4-a863-113939c64342-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "993ca158-d110-45c4-a863-113939c64342" (UID: "993ca158-d110-45c4-a863-113939c64342"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:13:18.486077 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.486058    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993ca158-d110-45c4-a863-113939c64342-kube-api-access-2sxg6" (OuterVolumeSpecName: "kube-api-access-2sxg6") pod "993ca158-d110-45c4-a863-113939c64342" (UID: "993ca158-d110-45c4-a863-113939c64342"). InnerVolumeSpecName "kube-api-access-2sxg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:13:18.585054 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.585028    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/993ca158-d110-45c4-a863-113939c64342-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:13:18.585054 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.585049    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/993ca158-d110-45c4-a863-113939c64342-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:13:18.585054 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.585059    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sxg6\" (UniqueName: \"kubernetes.io/projected/993ca158-d110-45c4-a863-113939c64342-kube-api-access-2sxg6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:13:18.585252 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.585068    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/993ca158-d110-45c4-a863-113939c64342-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:13:18.779841 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.779789    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"]
Sep 16 18:13:18.786306 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:18.786283    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-cd9cbcb46-28bh5"]
Sep 16 18:13:19.767064 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:19.767026    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="993ca158-d110-45c4-a863-113939c64342" path="/var/lib/kubelet/pods/993ca158-d110-45c4-a863-113939c64342/volumes"
Sep 16 18:13:52.502048 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:52.502017    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:13:53.579083 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.579048    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:13:53.610885 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.610841    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"]
Sep 16 18:13:53.611286 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.611267    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kserve-container"
Sep 16 18:13:53.611378 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.611289    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kserve-container"
Sep 16 18:13:53.611378 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.611306    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="storage-initializer"
Sep 16 18:13:53.611378 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.611314    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="storage-initializer"
Sep 16 18:13:53.611378 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.611323    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kserve-agent"
Sep 16 18:13:53.611378 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.611330    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kserve-agent"
Sep 16 18:13:53.611378 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.611374    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kube-rbac-proxy"
Sep 16 18:13:53.611689 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.611383    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kube-rbac-proxy"
Sep 16 18:13:53.611689 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.611459    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kube-rbac-proxy"
Sep 16 18:13:53.611689 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.611474    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kserve-agent"
Sep 16 18:13:53.611689 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.611484    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="993ca158-d110-45c4-a863-113939c64342" containerName="kserve-container"
Sep 16 18:13:53.614908 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.614867    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:13:53.617536 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.617508    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 18:13:53.617700 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.617576    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 16 18:13:53.617700 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.617598    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tnq4s\""
Sep 16 18:13:53.617700 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.617533    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 16 18:13:53.617700 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.617546    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:13:53.617700 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.617546    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:13:53.624586 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.624557    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"]
Sep 16 18:13:53.676093 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.676045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7gvv\" (UniqueName: \"kubernetes.io/projected/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kube-api-access-r7gvv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:13:53.676321 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.676108    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:13:53.676321 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.676133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:13:53.676321 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.676190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:13:53.777186 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.777145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:13:53.777344 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.777222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7gvv\" (UniqueName: \"kubernetes.io/projected/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kube-api-access-r7gvv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:13:53.777344 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.777284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:13:53.777344 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.777314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:13:53.777622 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.777596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:13:53.778008 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.777980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:13:53.779839 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.779815    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:13:53.789511 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.789480    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7gvv\" (UniqueName: \"kubernetes.io/projected/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kube-api-access-r7gvv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:13:53.818605 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.818568    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:13:53.925724 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:53.925620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:13:54.201185 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:54.201152    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"]
Sep 16 18:13:54.203337 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:13:54.203306    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea10aee_b9ca_4ab1_a440_2e69cdef3c2a.slice/crio-ecde35fc7c2eb5bd6883538808bca51fcbff5731e0051c61e744baecb3ab373a WatchSource:0}: Error finding container ecde35fc7c2eb5bd6883538808bca51fcbff5731e0051c61e744baecb3ab373a: Status 404 returned error can't find the container with id ecde35fc7c2eb5bd6883538808bca51fcbff5731e0051c61e744baecb3ab373a
Sep 16 18:13:54.607618 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:54.607581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d" event={"ID":"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a","Type":"ContainerStarted","Data":"bca9864b973694ab30e36f7aeff27af9f2400bdfac2b276ed2848a6060adeb22"}
Sep 16 18:13:54.608041 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:13:54.607641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d" event={"ID":"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a","Type":"ContainerStarted","Data":"ecde35fc7c2eb5bd6883538808bca51fcbff5731e0051c61e744baecb3ab373a"}
Sep 16 18:14:00.634855 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:00.634773    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d_8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a/storage-initializer/0.log"
Sep 16 18:14:00.634855 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:00.634818    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a" containerID="bca9864b973694ab30e36f7aeff27af9f2400bdfac2b276ed2848a6060adeb22" exitCode=1
Sep 16 18:14:00.635301 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:00.634877    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d" event={"ID":"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a","Type":"ContainerDied","Data":"bca9864b973694ab30e36f7aeff27af9f2400bdfac2b276ed2848a6060adeb22"}
Sep 16 18:14:01.640607 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:01.640580    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d_8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a/storage-initializer/0.log"
Sep 16 18:14:01.641008 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:01.640635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d" event={"ID":"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a","Type":"ContainerStarted","Data":"c6c94d675ac7a576208483ef55967aa5b847acfacf8d50255ed01412901c55db"}
Sep 16 18:14:03.609244 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:03.609210    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"]
Sep 16 18:14:03.609626 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:03.609416    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d" podUID="8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a" containerName="storage-initializer" containerID="cri-o://c6c94d675ac7a576208483ef55967aa5b847acfacf8d50255ed01412901c55db" gracePeriod=30
Sep 16 18:14:03.622311 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:03.622283    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:14:04.439609 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.439576    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:14:04.688457 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.688421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:14:04.728931 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.728830    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"]
Sep 16 18:14:04.734977 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.733216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:04.737644 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.737618    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 16 18:14:04.738293 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.738274    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:14:04.738867 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.738849    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 16 18:14:04.759625 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.759598    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"]
Sep 16 18:14:04.871256 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.871218    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82bb601a-82fa-4b00-a976-d7c39c79a8bf-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:04.871256 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.871258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:04.871498 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.871289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njzcv\" (UniqueName: \"kubernetes.io/projected/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kube-api-access-njzcv\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:04.871498 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.871304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:04.871498 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.871413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82bb601a-82fa-4b00-a976-d7c39c79a8bf-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:04.972260 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.972219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82bb601a-82fa-4b00-a976-d7c39c79a8bf-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:04.972260 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.972261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:04.972557 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.972293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njzcv\" (UniqueName: \"kubernetes.io/projected/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kube-api-access-njzcv\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:04.972557 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.972318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:04.972557 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.972355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82bb601a-82fa-4b00-a976-d7c39c79a8bf-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:04.972557 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:14:04.972411    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 16 18:14:04.972557 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:14:04.972490    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82bb601a-82fa-4b00-a976-d7c39c79a8bf-proxy-tls podName:82bb601a-82fa-4b00-a976-d7c39c79a8bf nodeName:}" failed. No retries permitted until 2026-09-16 18:14:05.47247135 +0000 UTC m=+2974.261093863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/82bb601a-82fa-4b00-a976-d7c39c79a8bf-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" (UID: "82bb601a-82fa-4b00-a976-d7c39c79a8bf") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 16 18:14:04.972981 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.972954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:04.973105 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.973080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:04.973105 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.973100    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82bb601a-82fa-4b00-a976-d7c39c79a8bf-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:04.982751 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:04.982680    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njzcv\" (UniqueName: \"kubernetes.io/projected/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kube-api-access-njzcv\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:05.138414 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.138353    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:14:05.250488 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.250463    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d_8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a/storage-initializer/1.log"
Sep 16 18:14:05.250910 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.250872    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d_8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a/storage-initializer/0.log"
Sep 16 18:14:05.253225 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.253202    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:14:05.374822 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.374788    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-proxy-tls\") pod \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") "
Sep 16 18:14:05.375078 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.374862    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7gvv\" (UniqueName: \"kubernetes.io/projected/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kube-api-access-r7gvv\") pod \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") "
Sep 16 18:14:05.375078 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.374922    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kserve-provision-location\") pod \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") "
Sep 16 18:14:05.375078 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.374961    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kube-rbac-proxy-sar-config\") pod \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\" (UID: \"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a\") "
Sep 16 18:14:05.375258 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.375214    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a" (UID: "8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:14:05.375374 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.375355    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a" (UID: "8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:14:05.377146 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.377119    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kube-api-access-r7gvv" (OuterVolumeSpecName: "kube-api-access-r7gvv") pod "8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a" (UID: "8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a"). InnerVolumeSpecName "kube-api-access-r7gvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:14:05.377247 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.377199    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a" (UID: "8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:14:05.476126 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.476076    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82bb601a-82fa-4b00-a976-d7c39c79a8bf-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:05.476311 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.476143    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:14:05.476311 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.476158    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:14:05.476311 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.476172    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7gvv\" (UniqueName: \"kubernetes.io/projected/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kube-api-access-r7gvv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:14:05.476311 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.476183    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:14:05.478640 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.478618    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82bb601a-82fa-4b00-a976-d7c39c79a8bf-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:05.646159 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.646120    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:05.659269 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.659246    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d_8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a/storage-initializer/1.log"
Sep 16 18:14:05.659649 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.659631    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d_8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a/storage-initializer/0.log"
Sep 16 18:14:05.659725 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.659669    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a" containerID="c6c94d675ac7a576208483ef55967aa5b847acfacf8d50255ed01412901c55db" exitCode=1
Sep 16 18:14:05.659769 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.659729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d" event={"ID":"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a","Type":"ContainerDied","Data":"c6c94d675ac7a576208483ef55967aa5b847acfacf8d50255ed01412901c55db"}
Sep 16 18:14:05.659769 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.659739    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"
Sep 16 18:14:05.659849 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.659791    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d" event={"ID":"8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a","Type":"ContainerDied","Data":"ecde35fc7c2eb5bd6883538808bca51fcbff5731e0051c61e744baecb3ab373a"}
Sep 16 18:14:05.659849 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.659828    2581 scope.go:117] "RemoveContainer" containerID="c6c94d675ac7a576208483ef55967aa5b847acfacf8d50255ed01412901c55db"
Sep 16 18:14:05.669541 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.669517    2581 scope.go:117] "RemoveContainer" containerID="bca9864b973694ab30e36f7aeff27af9f2400bdfac2b276ed2848a6060adeb22"
Sep 16 18:14:05.678303 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.678280    2581 scope.go:117] "RemoveContainer" containerID="c6c94d675ac7a576208483ef55967aa5b847acfacf8d50255ed01412901c55db"
Sep 16 18:14:05.678617 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:14:05.678598    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6c94d675ac7a576208483ef55967aa5b847acfacf8d50255ed01412901c55db\": container with ID starting with c6c94d675ac7a576208483ef55967aa5b847acfacf8d50255ed01412901c55db not found: ID does not exist" containerID="c6c94d675ac7a576208483ef55967aa5b847acfacf8d50255ed01412901c55db"
Sep 16 18:14:05.678697 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.678631    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6c94d675ac7a576208483ef55967aa5b847acfacf8d50255ed01412901c55db"} err="failed to get container status \"c6c94d675ac7a576208483ef55967aa5b847acfacf8d50255ed01412901c55db\": rpc error: code = NotFound desc = could not find container \"c6c94d675ac7a576208483ef55967aa5b847acfacf8d50255ed01412901c55db\": container with ID starting with c6c94d675ac7a576208483ef55967aa5b847acfacf8d50255ed01412901c55db not found: ID does not exist"
Sep 16 18:14:05.678697 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.678656    2581 scope.go:117] "RemoveContainer" containerID="bca9864b973694ab30e36f7aeff27af9f2400bdfac2b276ed2848a6060adeb22"
Sep 16 18:14:05.678984 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:14:05.678964    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca9864b973694ab30e36f7aeff27af9f2400bdfac2b276ed2848a6060adeb22\": container with ID starting with bca9864b973694ab30e36f7aeff27af9f2400bdfac2b276ed2848a6060adeb22 not found: ID does not exist" containerID="bca9864b973694ab30e36f7aeff27af9f2400bdfac2b276ed2848a6060adeb22"
Sep 16 18:14:05.679041 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.678990    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca9864b973694ab30e36f7aeff27af9f2400bdfac2b276ed2848a6060adeb22"} err="failed to get container status \"bca9864b973694ab30e36f7aeff27af9f2400bdfac2b276ed2848a6060adeb22\": rpc error: code = NotFound desc = could not find container \"bca9864b973694ab30e36f7aeff27af9f2400bdfac2b276ed2848a6060adeb22\": container with ID starting with bca9864b973694ab30e36f7aeff27af9f2400bdfac2b276ed2848a6060adeb22 not found: ID does not exist"
Sep 16 18:14:05.702063 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.702039    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"]
Sep 16 18:14:05.706454 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.706421    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6b6dfb74f9-m7q2d"]
Sep 16 18:14:05.765680 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.765645    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a" path="/var/lib/kubelet/pods/8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a/volumes"
Sep 16 18:14:05.777217 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.777186    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:14:05.839303 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.839275    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:14:05.931148 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:05.931123    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"]
Sep 16 18:14:05.933446 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:14:05.933419    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82bb601a_82fa_4b00_a976_d7c39c79a8bf.slice/crio-b1a92f41899c74bbffd224d1ca8244638150dd5d65e79fd8c83ceb638766b9a3 WatchSource:0}: Error finding container b1a92f41899c74bbffd224d1ca8244638150dd5d65e79fd8c83ceb638766b9a3: Status 404 returned error can't find the container with id b1a92f41899c74bbffd224d1ca8244638150dd5d65e79fd8c83ceb638766b9a3
Sep 16 18:14:06.666710 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:06.666675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" event={"ID":"82bb601a-82fa-4b00-a976-d7c39c79a8bf","Type":"ContainerStarted","Data":"76c255b944bbc2dbf01b6c48fbaf5008af3e78e353875b14db67ee66fa5aa042"}
Sep 16 18:14:06.666956 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:06.666738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" event={"ID":"82bb601a-82fa-4b00-a976-d7c39c79a8bf","Type":"ContainerStarted","Data":"b1a92f41899c74bbffd224d1ca8244638150dd5d65e79fd8c83ceb638766b9a3"}
Sep 16 18:14:07.672834 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:07.672802    2581 generic.go:356] "Generic (PLEG): container finished" podID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerID="76c255b944bbc2dbf01b6c48fbaf5008af3e78e353875b14db67ee66fa5aa042" exitCode=0
Sep 16 18:14:07.673247 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:07.672859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" event={"ID":"82bb601a-82fa-4b00-a976-d7c39c79a8bf","Type":"ContainerDied","Data":"76c255b944bbc2dbf01b6c48fbaf5008af3e78e353875b14db67ee66fa5aa042"}
Sep 16 18:14:08.684840 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:08.684799    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" event={"ID":"82bb601a-82fa-4b00-a976-d7c39c79a8bf","Type":"ContainerStarted","Data":"ead25aac6a12a881fd029cc6f0a3b3dcc73ee5cbd2175fe38f2e2a71046189d6"}
Sep 16 18:14:08.685281 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:08.684886    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:08.685281 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:08.685024    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:08.685281 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:08.685077    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" event={"ID":"82bb601a-82fa-4b00-a976-d7c39c79a8bf","Type":"ContainerStarted","Data":"0dca28c17041b136cdfa4cffb8d58e849a23f654f6febde5026a3695881f6c0f"}
Sep 16 18:14:08.686452 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:08.686426    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 18:14:09.582423 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:09.582396    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:14:09.689831 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:09.689793    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 18:14:10.978715 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:10.978682    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:14:14.694987 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:14.694953    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:14:14.695549 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:14.695524    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 18:14:14.719492 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:14.719447    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" podStartSLOduration=9.575840356 podStartE2EDuration="10.719432366s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.14359201s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:14:08.717702537 +0000 UTC m=+2977.506325081" watchObservedRunningTime="2026-09-16 18:14:14.719432366 +0000 UTC m=+2983.508054898"
Sep 16 18:14:24.696487 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:24.696446    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 18:14:31.919054 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:31.919024    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 18:14:31.921955 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:31.921930    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 18:14:34.696372 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:34.696324    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 18:14:44.696156 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:44.696111    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 18:14:54.696225 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:14:54.696183    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 18:15:04.695938 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:04.695883    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 18:15:10.785437 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:10.785401    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:14.695690 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:14.695653    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 16 18:15:15.973608 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:15.973576    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:16.464799 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:16.464764    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:24.696666 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:24.696595    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:15:34.724556 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:34.724526    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"]
Sep 16 18:15:34.725039 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:34.724751    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kserve-container" containerID="cri-o://0dca28c17041b136cdfa4cffb8d58e849a23f654f6febde5026a3695881f6c0f" gracePeriod=30
Sep 16 18:15:34.725039 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:34.724798    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kube-rbac-proxy" containerID="cri-o://ead25aac6a12a881fd029cc6f0a3b3dcc73ee5cbd2175fe38f2e2a71046189d6" gracePeriod=30
Sep 16 18:15:34.739470 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:34.739450    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:35.030661 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.030632    2581 generic.go:356] "Generic (PLEG): container finished" podID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerID="ead25aac6a12a881fd029cc6f0a3b3dcc73ee5cbd2175fe38f2e2a71046189d6" exitCode=2
Sep 16 18:15:35.030786 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.030674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" event={"ID":"82bb601a-82fa-4b00-a976-d7c39c79a8bf","Type":"ContainerDied","Data":"ead25aac6a12a881fd029cc6f0a3b3dcc73ee5cbd2175fe38f2e2a71046189d6"}
Sep 16 18:15:35.817151 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.817123    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:35.843724 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.843695    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"]
Sep 16 18:15:35.844019 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.844006    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a" containerName="storage-initializer"
Sep 16 18:15:35.844075 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.844021    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a" containerName="storage-initializer"
Sep 16 18:15:35.844111 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.844088    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a" containerName="storage-initializer"
Sep 16 18:15:35.844160 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.844150    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a" containerName="storage-initializer"
Sep 16 18:15:35.844194 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.844159    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a" containerName="storage-initializer"
Sep 16 18:15:35.844227 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.844202    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ea10aee-b9ca-4ab1-a440-2e69cdef3c2a" containerName="storage-initializer"
Sep 16 18:15:35.847150 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.847133    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:35.849770 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.849751    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 16 18:15:35.849878 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.849772    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 16 18:15:35.864341 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.864321    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"]
Sep 16 18:15:35.923219 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.923184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:35.923395 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.923226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1fc1057-d977-4e52-a9e7-6fc650799a5c-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:35.923395 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.923246    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs9nx\" (UniqueName: \"kubernetes.io/projected/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kube-api-access-vs9nx\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:35.923395 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:35.923309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:36.003963 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:36.003933    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:36.024192 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:36.024165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:36.024276 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:36.024203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:36.024347 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:36.024321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1fc1057-d977-4e52-a9e7-6fc650799a5c-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:36.024409 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:36.024351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs9nx\" (UniqueName: \"kubernetes.io/projected/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kube-api-access-vs9nx\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:36.024612 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:36.024591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:36.024958 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:36.024939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:36.026940 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:36.026919    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1fc1057-d977-4e52-a9e7-6fc650799a5c-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:36.032236 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:36.032218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs9nx\" (UniqueName: \"kubernetes.io/projected/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kube-api-access-vs9nx\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:36.156785 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:36.156700    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:36.433424 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:36.433346    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"]
Sep 16 18:15:36.436921 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:15:36.436871    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fc1057_d977_4e52_a9e7_6fc650799a5c.slice/crio-4247efbbc3ac5091b62e5ee9545a17585ba5ecc8ea751d0db2bb7e62ff287382 WatchSource:0}: Error finding container 4247efbbc3ac5091b62e5ee9545a17585ba5ecc8ea751d0db2bb7e62ff287382: Status 404 returned error can't find the container with id 4247efbbc3ac5091b62e5ee9545a17585ba5ecc8ea751d0db2bb7e62ff287382
Sep 16 18:15:37.041078 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:37.041041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2" event={"ID":"c1fc1057-d977-4e52-a9e7-6fc650799a5c","Type":"ContainerStarted","Data":"bccd2cb11e7fc3da753c0ab9f3efe7b8ea53a466e0fa638b866cb05796adde30"}
Sep 16 18:15:37.041421 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:37.041111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2" event={"ID":"c1fc1057-d977-4e52-a9e7-6fc650799a5c","Type":"ContainerStarted","Data":"4247efbbc3ac5091b62e5ee9545a17585ba5ecc8ea751d0db2bb7e62ff287382"}
Sep 16 18:15:39.051488 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.051459    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2_c1fc1057-d977-4e52-a9e7-6fc650799a5c/storage-initializer/0.log"
Sep 16 18:15:39.051952 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.051501    2581 generic.go:356] "Generic (PLEG): container finished" podID="c1fc1057-d977-4e52-a9e7-6fc650799a5c" containerID="bccd2cb11e7fc3da753c0ab9f3efe7b8ea53a466e0fa638b866cb05796adde30" exitCode=1
Sep 16 18:15:39.051952 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.051547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2" event={"ID":"c1fc1057-d977-4e52-a9e7-6fc650799a5c","Type":"ContainerDied","Data":"bccd2cb11e7fc3da753c0ab9f3efe7b8ea53a466e0fa638b866cb05796adde30"}
Sep 16 18:15:39.369369 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.369343    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:15:39.451340 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.451308    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njzcv\" (UniqueName: \"kubernetes.io/projected/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kube-api-access-njzcv\") pod \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") "
Sep 16 18:15:39.451527 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.451372    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kserve-provision-location\") pod \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") "
Sep 16 18:15:39.451527 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.451441    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82bb601a-82fa-4b00-a976-d7c39c79a8bf-cabundle-cert\") pod \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") "
Sep 16 18:15:39.451527 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.451470    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kube-rbac-proxy-sar-config\") pod \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") "
Sep 16 18:15:39.451689 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.451532    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82bb601a-82fa-4b00-a976-d7c39c79a8bf-proxy-tls\") pod \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\" (UID: \"82bb601a-82fa-4b00-a976-d7c39c79a8bf\") "
Sep 16 18:15:39.451803 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.451751    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "82bb601a-82fa-4b00-a976-d7c39c79a8bf" (UID: "82bb601a-82fa-4b00-a976-d7c39c79a8bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:15:39.451987 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.451819    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82bb601a-82fa-4b00-a976-d7c39c79a8bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "82bb601a-82fa-4b00-a976-d7c39c79a8bf" (UID: "82bb601a-82fa-4b00-a976-d7c39c79a8bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:15:39.451987 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.451836    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "82bb601a-82fa-4b00-a976-d7c39c79a8bf" (UID: "82bb601a-82fa-4b00-a976-d7c39c79a8bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:15:39.453913 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.453875    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82bb601a-82fa-4b00-a976-d7c39c79a8bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "82bb601a-82fa-4b00-a976-d7c39c79a8bf" (UID: "82bb601a-82fa-4b00-a976-d7c39c79a8bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:15:39.454077 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.454056    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kube-api-access-njzcv" (OuterVolumeSpecName: "kube-api-access-njzcv") pod "82bb601a-82fa-4b00-a976-d7c39c79a8bf" (UID: "82bb601a-82fa-4b00-a976-d7c39c79a8bf"). InnerVolumeSpecName "kube-api-access-njzcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:15:39.552500 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.552468    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82bb601a-82fa-4b00-a976-d7c39c79a8bf-cabundle-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:15:39.552500 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.552492    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:15:39.552500 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.552507    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82bb601a-82fa-4b00-a976-d7c39c79a8bf-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:15:39.552817 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.552518    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njzcv\" (UniqueName: \"kubernetes.io/projected/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kube-api-access-njzcv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:15:39.552817 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:39.552527    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82bb601a-82fa-4b00-a976-d7c39c79a8bf-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:15:40.057496 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.057464    2581 generic.go:356] "Generic (PLEG): container finished" podID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerID="0dca28c17041b136cdfa4cffb8d58e849a23f654f6febde5026a3695881f6c0f" exitCode=0
Sep 16 18:15:40.057905 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.057504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" event={"ID":"82bb601a-82fa-4b00-a976-d7c39c79a8bf","Type":"ContainerDied","Data":"0dca28c17041b136cdfa4cffb8d58e849a23f654f6febde5026a3695881f6c0f"}
Sep 16 18:15:40.057905 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.057528    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"
Sep 16 18:15:40.057905 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.057602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp" event={"ID":"82bb601a-82fa-4b00-a976-d7c39c79a8bf","Type":"ContainerDied","Data":"b1a92f41899c74bbffd224d1ca8244638150dd5d65e79fd8c83ceb638766b9a3"}
Sep 16 18:15:40.057905 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.057638    2581 scope.go:117] "RemoveContainer" containerID="ead25aac6a12a881fd029cc6f0a3b3dcc73ee5cbd2175fe38f2e2a71046189d6"
Sep 16 18:15:40.059781 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.059763    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2_c1fc1057-d977-4e52-a9e7-6fc650799a5c/storage-initializer/0.log"
Sep 16 18:15:40.059958 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.059824    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2" event={"ID":"c1fc1057-d977-4e52-a9e7-6fc650799a5c","Type":"ContainerStarted","Data":"651a89bbe6388a264496dd7b853d6941d3a57bc4d94ad697f4972ec13d6a3548"}
Sep 16 18:15:40.066350 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.066332    2581 scope.go:117] "RemoveContainer" containerID="0dca28c17041b136cdfa4cffb8d58e849a23f654f6febde5026a3695881f6c0f"
Sep 16 18:15:40.073828 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.073813    2581 scope.go:117] "RemoveContainer" containerID="76c255b944bbc2dbf01b6c48fbaf5008af3e78e353875b14db67ee66fa5aa042"
Sep 16 18:15:40.080769 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.080753    2581 scope.go:117] "RemoveContainer" containerID="ead25aac6a12a881fd029cc6f0a3b3dcc73ee5cbd2175fe38f2e2a71046189d6"
Sep 16 18:15:40.081051 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:15:40.081033    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead25aac6a12a881fd029cc6f0a3b3dcc73ee5cbd2175fe38f2e2a71046189d6\": container with ID starting with ead25aac6a12a881fd029cc6f0a3b3dcc73ee5cbd2175fe38f2e2a71046189d6 not found: ID does not exist" containerID="ead25aac6a12a881fd029cc6f0a3b3dcc73ee5cbd2175fe38f2e2a71046189d6"
Sep 16 18:15:40.081110 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.081059    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead25aac6a12a881fd029cc6f0a3b3dcc73ee5cbd2175fe38f2e2a71046189d6"} err="failed to get container status \"ead25aac6a12a881fd029cc6f0a3b3dcc73ee5cbd2175fe38f2e2a71046189d6\": rpc error: code = NotFound desc = could not find container \"ead25aac6a12a881fd029cc6f0a3b3dcc73ee5cbd2175fe38f2e2a71046189d6\": container with ID starting with ead25aac6a12a881fd029cc6f0a3b3dcc73ee5cbd2175fe38f2e2a71046189d6 not found: ID does not exist"
Sep 16 18:15:40.081110 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.081075    2581 scope.go:117] "RemoveContainer" containerID="0dca28c17041b136cdfa4cffb8d58e849a23f654f6febde5026a3695881f6c0f"
Sep 16 18:15:40.081302 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:15:40.081286    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dca28c17041b136cdfa4cffb8d58e849a23f654f6febde5026a3695881f6c0f\": container with ID starting with 0dca28c17041b136cdfa4cffb8d58e849a23f654f6febde5026a3695881f6c0f not found: ID does not exist" containerID="0dca28c17041b136cdfa4cffb8d58e849a23f654f6febde5026a3695881f6c0f"
Sep 16 18:15:40.081344 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.081306    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dca28c17041b136cdfa4cffb8d58e849a23f654f6febde5026a3695881f6c0f"} err="failed to get container status \"0dca28c17041b136cdfa4cffb8d58e849a23f654f6febde5026a3695881f6c0f\": rpc error: code = NotFound desc = could not find container \"0dca28c17041b136cdfa4cffb8d58e849a23f654f6febde5026a3695881f6c0f\": container with ID starting with 0dca28c17041b136cdfa4cffb8d58e849a23f654f6febde5026a3695881f6c0f not found: ID does not exist"
Sep 16 18:15:40.081344 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.081318    2581 scope.go:117] "RemoveContainer" containerID="76c255b944bbc2dbf01b6c48fbaf5008af3e78e353875b14db67ee66fa5aa042"
Sep 16 18:15:40.081516 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:15:40.081498    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c255b944bbc2dbf01b6c48fbaf5008af3e78e353875b14db67ee66fa5aa042\": container with ID starting with 76c255b944bbc2dbf01b6c48fbaf5008af3e78e353875b14db67ee66fa5aa042 not found: ID does not exist" containerID="76c255b944bbc2dbf01b6c48fbaf5008af3e78e353875b14db67ee66fa5aa042"
Sep 16 18:15:40.081560 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.081520    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c255b944bbc2dbf01b6c48fbaf5008af3e78e353875b14db67ee66fa5aa042"} err="failed to get container status \"76c255b944bbc2dbf01b6c48fbaf5008af3e78e353875b14db67ee66fa5aa042\": rpc error: code = NotFound desc = could not find container \"76c255b944bbc2dbf01b6c48fbaf5008af3e78e353875b14db67ee66fa5aa042\": container with ID starting with 76c255b944bbc2dbf01b6c48fbaf5008af3e78e353875b14db67ee66fa5aa042 not found: ID does not exist"
Sep 16 18:15:40.101770 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.101724    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"]
Sep 16 18:15:40.105130 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:40.105108    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-t9tfp"]
Sep 16 18:15:41.766206 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:41.766175    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" path="/var/lib/kubelet/pods/82bb601a-82fa-4b00-a976-d7c39c79a8bf/volumes"
Sep 16 18:15:45.844191 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:45.844161    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"]
Sep 16 18:15:45.844700 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:45.844373    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2" podUID="c1fc1057-d977-4e52-a9e7-6fc650799a5c" containerName="storage-initializer" containerID="cri-o://651a89bbe6388a264496dd7b853d6941d3a57bc4d94ad697f4972ec13d6a3548" gracePeriod=30
Sep 16 18:15:45.866027 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:45.865993    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:45.972295 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:45.972275    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2_c1fc1057-d977-4e52-a9e7-6fc650799a5c/storage-initializer/1.log"
Sep 16 18:15:45.972707 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:45.972692    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2_c1fc1057-d977-4e52-a9e7-6fc650799a5c/storage-initializer/0.log"
Sep 16 18:15:45.974985 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:45.974969    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:46.002624 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.002598    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs9nx\" (UniqueName: \"kubernetes.io/projected/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kube-api-access-vs9nx\") pod \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") "
Sep 16 18:15:46.002760 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.002641    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kube-rbac-proxy-sar-config\") pod \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") "
Sep 16 18:15:46.002760 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.002728    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1fc1057-d977-4e52-a9e7-6fc650799a5c-proxy-tls\") pod \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") "
Sep 16 18:15:46.002760 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.002754    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kserve-provision-location\") pod \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\" (UID: \"c1fc1057-d977-4e52-a9e7-6fc650799a5c\") "
Sep 16 18:15:46.003083 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.003057    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1fc1057-d977-4e52-a9e7-6fc650799a5c" (UID: "c1fc1057-d977-4e52-a9e7-6fc650799a5c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:15:46.003163 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.003085    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c1fc1057-d977-4e52-a9e7-6fc650799a5c" (UID: "c1fc1057-d977-4e52-a9e7-6fc650799a5c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:15:46.004856 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.004837    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kube-api-access-vs9nx" (OuterVolumeSpecName: "kube-api-access-vs9nx") pod "c1fc1057-d977-4e52-a9e7-6fc650799a5c" (UID: "c1fc1057-d977-4e52-a9e7-6fc650799a5c"). InnerVolumeSpecName "kube-api-access-vs9nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:15:46.004942 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.004870    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1fc1057-d977-4e52-a9e7-6fc650799a5c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1fc1057-d977-4e52-a9e7-6fc650799a5c" (UID: "c1fc1057-d977-4e52-a9e7-6fc650799a5c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:15:46.085959 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.085879    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2_c1fc1057-d977-4e52-a9e7-6fc650799a5c/storage-initializer/1.log"
Sep 16 18:15:46.086237 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.086225    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2_c1fc1057-d977-4e52-a9e7-6fc650799a5c/storage-initializer/0.log"
Sep 16 18:15:46.086288 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.086255    2581 generic.go:356] "Generic (PLEG): container finished" podID="c1fc1057-d977-4e52-a9e7-6fc650799a5c" containerID="651a89bbe6388a264496dd7b853d6941d3a57bc4d94ad697f4972ec13d6a3548" exitCode=1
Sep 16 18:15:46.086340 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.086315    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"
Sep 16 18:15:46.086340 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.086332    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2" event={"ID":"c1fc1057-d977-4e52-a9e7-6fc650799a5c","Type":"ContainerDied","Data":"651a89bbe6388a264496dd7b853d6941d3a57bc4d94ad697f4972ec13d6a3548"}
Sep 16 18:15:46.086420 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.086387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2" event={"ID":"c1fc1057-d977-4e52-a9e7-6fc650799a5c","Type":"ContainerDied","Data":"4247efbbc3ac5091b62e5ee9545a17585ba5ecc8ea751d0db2bb7e62ff287382"}
Sep 16 18:15:46.086420 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.086413    2581 scope.go:117] "RemoveContainer" containerID="651a89bbe6388a264496dd7b853d6941d3a57bc4d94ad697f4972ec13d6a3548"
Sep 16 18:15:46.097686 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.097594    2581 scope.go:117] "RemoveContainer" containerID="bccd2cb11e7fc3da753c0ab9f3efe7b8ea53a466e0fa638b866cb05796adde30"
Sep 16 18:15:46.103272 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.103249    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:15:46.103272 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.103268    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1fc1057-d977-4e52-a9e7-6fc650799a5c-proxy-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:15:46.103431 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.103279    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kserve-provision-location\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:15:46.103431 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.103287    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs9nx\" (UniqueName: \"kubernetes.io/projected/c1fc1057-d977-4e52-a9e7-6fc650799a5c-kube-api-access-vs9nx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:15:46.106088 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.106074    2581 scope.go:117] "RemoveContainer" containerID="651a89bbe6388a264496dd7b853d6941d3a57bc4d94ad697f4972ec13d6a3548"
Sep 16 18:15:46.106337 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:15:46.106321    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651a89bbe6388a264496dd7b853d6941d3a57bc4d94ad697f4972ec13d6a3548\": container with ID starting with 651a89bbe6388a264496dd7b853d6941d3a57bc4d94ad697f4972ec13d6a3548 not found: ID does not exist" containerID="651a89bbe6388a264496dd7b853d6941d3a57bc4d94ad697f4972ec13d6a3548"
Sep 16 18:15:46.106385 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.106344    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651a89bbe6388a264496dd7b853d6941d3a57bc4d94ad697f4972ec13d6a3548"} err="failed to get container status \"651a89bbe6388a264496dd7b853d6941d3a57bc4d94ad697f4972ec13d6a3548\": rpc error: code = NotFound desc = could not find container \"651a89bbe6388a264496dd7b853d6941d3a57bc4d94ad697f4972ec13d6a3548\": container with ID starting with 651a89bbe6388a264496dd7b853d6941d3a57bc4d94ad697f4972ec13d6a3548 not found: ID does not exist"
Sep 16 18:15:46.106385 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.106359    2581 scope.go:117] "RemoveContainer" containerID="bccd2cb11e7fc3da753c0ab9f3efe7b8ea53a466e0fa638b866cb05796adde30"
Sep 16 18:15:46.106613 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:15:46.106595    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bccd2cb11e7fc3da753c0ab9f3efe7b8ea53a466e0fa638b866cb05796adde30\": container with ID starting with bccd2cb11e7fc3da753c0ab9f3efe7b8ea53a466e0fa638b866cb05796adde30 not found: ID does not exist" containerID="bccd2cb11e7fc3da753c0ab9f3efe7b8ea53a466e0fa638b866cb05796adde30"
Sep 16 18:15:46.106657 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.106618    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bccd2cb11e7fc3da753c0ab9f3efe7b8ea53a466e0fa638b866cb05796adde30"} err="failed to get container status \"bccd2cb11e7fc3da753c0ab9f3efe7b8ea53a466e0fa638b866cb05796adde30\": rpc error: code = NotFound desc = could not find container \"bccd2cb11e7fc3da753c0ab9f3efe7b8ea53a466e0fa638b866cb05796adde30\": container with ID starting with bccd2cb11e7fc3da753c0ab9f3efe7b8ea53a466e0fa638b866cb05796adde30 not found: ID does not exist"
Sep 16 18:15:46.143048 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.143021    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"]
Sep 16 18:15:46.145316 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.145294    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7f54548f9d-tmrz2"]
Sep 16 18:15:46.941083 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:46.941052    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:47.130706 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:47.130671    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:47.766386 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:15:47.766355    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1fc1057-d977-4e52-a9e7-6fc650799a5c" path="/var/lib/kubelet/pods/c1fc1057-d977-4e52-a9e7-6fc650799a5c/volumes"
Sep 16 18:16:17.340329 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:16:17.340298    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:22.503355 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:16:22.503325    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:22.744059 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:16:22.744025    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:28.325758 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:16:28.325727    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:17.913874 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:17:17.913798    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:28.139560 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:17:28.139531    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:33.284859 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:17:33.284829    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:33.488837 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:17:33.488808    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:53.835396 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:17:53.835364    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:30.458484 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:18:30.458416    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:14.360360 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:19:14.360332    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:19.475995 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:19:19.475956    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:19.583573 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:19:19.583547    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:31.946373 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:19:31.946349    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 18:19:31.954081 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:19:31.954057    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 18:20:28.483581 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:20:28.483548    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:04.703009 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:04.702978    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:04.875118 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:04.875091    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:24.164487 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:24.164415    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:24.257301 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:24.257275    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:28.806904 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.806867    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sk7zf/must-gather-dlbdm"]
Sep 16 18:21:28.807248 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.807149    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="storage-initializer"
Sep 16 18:21:28.807248 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.807160    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="storage-initializer"
Sep 16 18:21:28.807248 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.807173    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kserve-container"
Sep 16 18:21:28.807248 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.807178    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kserve-container"
Sep 16 18:21:28.807248 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.807184    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1fc1057-d977-4e52-a9e7-6fc650799a5c" containerName="storage-initializer"
Sep 16 18:21:28.807248 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.807190    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1fc1057-d977-4e52-a9e7-6fc650799a5c" containerName="storage-initializer"
Sep 16 18:21:28.807248 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.807205    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1fc1057-d977-4e52-a9e7-6fc650799a5c" containerName="storage-initializer"
Sep 16 18:21:28.807248 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.807209    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1fc1057-d977-4e52-a9e7-6fc650799a5c" containerName="storage-initializer"
Sep 16 18:21:28.807248 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.807216    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kube-rbac-proxy"
Sep 16 18:21:28.807248 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.807222    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kube-rbac-proxy"
Sep 16 18:21:28.807543 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.807266    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kube-rbac-proxy"
Sep 16 18:21:28.807543 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.807276    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="82bb601a-82fa-4b00-a976-d7c39c79a8bf" containerName="kserve-container"
Sep 16 18:21:28.807543 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.807283    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1fc1057-d977-4e52-a9e7-6fc650799a5c" containerName="storage-initializer"
Sep 16 18:21:28.807543 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.807288    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1fc1057-d977-4e52-a9e7-6fc650799a5c" containerName="storage-initializer"
Sep 16 18:21:28.810269 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.810249    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sk7zf/must-gather-dlbdm"
Sep 16 18:21:28.811231 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.811203    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:28.812938 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.812918    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sk7zf\"/\"kube-root-ca.crt\""
Sep 16 18:21:28.813040 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.812922    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sk7zf\"/\"openshift-service-ca.crt\""
Sep 16 18:21:28.820279 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.820257    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sk7zf/must-gather-dlbdm"]
Sep 16 18:21:28.935042 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.935017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0942b983-9609-4de3-9896-7bad41758aef-must-gather-output\") pod \"must-gather-dlbdm\" (UID: \"0942b983-9609-4de3-9896-7bad41758aef\") " pod="openshift-must-gather-sk7zf/must-gather-dlbdm"
Sep 16 18:21:28.935128 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:28.935043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g88h\" (UniqueName: \"kubernetes.io/projected/0942b983-9609-4de3-9896-7bad41758aef-kube-api-access-4g88h\") pod \"must-gather-dlbdm\" (UID: \"0942b983-9609-4de3-9896-7bad41758aef\") " pod="openshift-must-gather-sk7zf/must-gather-dlbdm"
Sep 16 18:21:29.035535 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:29.035512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0942b983-9609-4de3-9896-7bad41758aef-must-gather-output\") pod \"must-gather-dlbdm\" (UID: \"0942b983-9609-4de3-9896-7bad41758aef\") " pod="openshift-must-gather-sk7zf/must-gather-dlbdm"
Sep 16 18:21:29.035619 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:29.035537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g88h\" (UniqueName: \"kubernetes.io/projected/0942b983-9609-4de3-9896-7bad41758aef-kube-api-access-4g88h\") pod \"must-gather-dlbdm\" (UID: \"0942b983-9609-4de3-9896-7bad41758aef\") " pod="openshift-must-gather-sk7zf/must-gather-dlbdm"
Sep 16 18:21:29.035818 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:29.035803    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0942b983-9609-4de3-9896-7bad41758aef-must-gather-output\") pod \"must-gather-dlbdm\" (UID: \"0942b983-9609-4de3-9896-7bad41758aef\") " pod="openshift-must-gather-sk7zf/must-gather-dlbdm"
Sep 16 18:21:29.043852 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:29.043834    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g88h\" (UniqueName: \"kubernetes.io/projected/0942b983-9609-4de3-9896-7bad41758aef-kube-api-access-4g88h\") pod \"must-gather-dlbdm\" (UID: \"0942b983-9609-4de3-9896-7bad41758aef\") " pod="openshift-must-gather-sk7zf/must-gather-dlbdm"
Sep 16 18:21:29.132318 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:29.132270    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sk7zf/must-gather-dlbdm"
Sep 16 18:21:29.212351 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:29.212323    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:29.401205 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:29.401148    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sk7zf/must-gather-dlbdm"]
Sep 16 18:21:29.404445 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:21:29.404423    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0942b983_9609_4de3_9896_7bad41758aef.slice/crio-e86a3d80340b03d867d10b45cad8af9b6c2b438e5daa5180a7c5b35d725aca0d WatchSource:0}: Error finding container e86a3d80340b03d867d10b45cad8af9b6c2b438e5daa5180a7c5b35d725aca0d: Status 404 returned error can't find the container with id e86a3d80340b03d867d10b45cad8af9b6c2b438e5daa5180a7c5b35d725aca0d
Sep 16 18:21:29.406102 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:29.406085    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:21:30.368954 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:30.368874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sk7zf/must-gather-dlbdm" event={"ID":"0942b983-9609-4de3-9896-7bad41758aef","Type":"ContainerStarted","Data":"e86a3d80340b03d867d10b45cad8af9b6c2b438e5daa5180a7c5b35d725aca0d"}
Sep 16 18:21:32.103456 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:32.103421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:35.396543 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:35.396425    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sk7zf/must-gather-dlbdm" event={"ID":"0942b983-9609-4de3-9896-7bad41758aef","Type":"ContainerStarted","Data":"71d53674699bb4e35d1116a91d9d0897674f139e41db5ddc9525e28f74de8887"}
Sep 16 18:21:36.402706 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:36.402673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sk7zf/must-gather-dlbdm" event={"ID":"0942b983-9609-4de3-9896-7bad41758aef","Type":"ContainerStarted","Data":"b9007750462cdd840c2486bef831aaeeb9a3a8d05ad1518cf0c1726108ec5153"}
Sep 16 18:21:36.420146 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:21:36.420095    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sk7zf/must-gather-dlbdm" podStartSLOduration=2.655106284 podStartE2EDuration="8.420081206s" totalImagesPullingTime="5.764974922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:21:36.418512231 +0000 UTC m=+3425.207134764" watchObservedRunningTime="2026-09-16 18:21:36.420081206 +0000 UTC m=+3425.208703737"
Sep 16 18:22:01.511284 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:01.511255    2581 generic.go:356] "Generic (PLEG): container finished" podID="0942b983-9609-4de3-9896-7bad41758aef" containerID="71d53674699bb4e35d1116a91d9d0897674f139e41db5ddc9525e28f74de8887" exitCode=0
Sep 16 18:22:01.511637 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:01.511299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sk7zf/must-gather-dlbdm" event={"ID":"0942b983-9609-4de3-9896-7bad41758aef","Type":"ContainerDied","Data":"71d53674699bb4e35d1116a91d9d0897674f139e41db5ddc9525e28f74de8887"}
Sep 16 18:22:01.511637 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:01.511527    2581 scope.go:117] "RemoveContainer" containerID="71d53674699bb4e35d1116a91d9d0897674f139e41db5ddc9525e28f74de8887"
Sep 16 18:22:01.798681 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:01.798659    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sk7zf_must-gather-dlbdm_0942b983-9609-4de3-9896-7bad41758aef/gather/0.log"
Sep 16 18:22:02.342721 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:02.342695    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:22:02.494227 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:02.494201    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:22:02.595497 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:02.595441    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:22:05.269693 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:05.269660    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jxft6_80100b19-7ac7-4382-af81-3f0a97bd7072/global-pull-secret-syncer/0.log"
Sep 16 18:22:05.383985 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:05.383961    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log"
Sep 16 18:22:05.481153 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:05.481137    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-31.ec2.internal_c40b89e27f34846eb7c4becaeceb3ee5/haproxy/0.log"
Sep 16 18:22:07.332541 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:07.332510    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-sk7zf/must-gather-dlbdm"]
Sep 16 18:22:07.332989 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:07.332718    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-sk7zf/must-gather-dlbdm" podUID="0942b983-9609-4de3-9896-7bad41758aef" containerName="copy" containerID="cri-o://b9007750462cdd840c2486bef831aaeeb9a3a8d05ad1518cf0c1726108ec5153" gracePeriod=2
Sep 16 18:22:07.335194 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:22:07.335165    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-dlbdm\" is forbidden: User \"system:node:ip-10-0-128-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sk7zf\": no relationship found between node 'ip-10-0-128-31.ec2.internal' and this object" podUID="0942b983-9609-4de3-9896-7bad41758aef" pod="openshift-must-gather-sk7zf/must-gather-dlbdm"
Sep 16 18:22:07.336649 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:07.336629    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-sk7zf/must-gather-dlbdm"]
Sep 16 18:22:07.535301 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:07.535280    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sk7zf_must-gather-dlbdm_0942b983-9609-4de3-9896-7bad41758aef/copy/0.log"
Sep 16 18:22:07.535689 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:07.535665    2581 generic.go:356] "Generic (PLEG): container finished" podID="0942b983-9609-4de3-9896-7bad41758aef" containerID="b9007750462cdd840c2486bef831aaeeb9a3a8d05ad1518cf0c1726108ec5153" exitCode=143
Sep 16 18:22:07.563188 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:07.563172    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sk7zf_must-gather-dlbdm_0942b983-9609-4de3-9896-7bad41758aef/copy/0.log"
Sep 16 18:22:07.565669 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:07.565652    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sk7zf/must-gather-dlbdm"
Sep 16 18:22:07.567811 ip-10-0-128-31 kubenswrapper[2581]: E0916 18:22:07.567790    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-dlbdm\" is forbidden: User \"system:node:ip-10-0-128-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sk7zf\": no relationship found between node 'ip-10-0-128-31.ec2.internal' and this object" podUID="0942b983-9609-4de3-9896-7bad41758aef" pod="openshift-must-gather-sk7zf/must-gather-dlbdm"
Sep 16 18:22:07.638460 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:07.638414    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0942b983-9609-4de3-9896-7bad41758aef-must-gather-output\") pod \"0942b983-9609-4de3-9896-7bad41758aef\" (UID: \"0942b983-9609-4de3-9896-7bad41758aef\") "
Sep 16 18:22:07.638529 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:07.638470    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g88h\" (UniqueName: \"kubernetes.io/projected/0942b983-9609-4de3-9896-7bad41758aef-kube-api-access-4g88h\") pod \"0942b983-9609-4de3-9896-7bad41758aef\" (UID: \"0942b983-9609-4de3-9896-7bad41758aef\") "
Sep 16 18:22:07.639456 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:07.639435    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0942b983-9609-4de3-9896-7bad41758aef-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "0942b983-9609-4de3-9896-7bad41758aef" (UID: "0942b983-9609-4de3-9896-7bad41758aef"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:22:07.640536 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:07.640517    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0942b983-9609-4de3-9896-7bad41758aef-kube-api-access-4g88h" (OuterVolumeSpecName: "kube-api-access-4g88h") pod "0942b983-9609-4de3-9896-7bad41758aef" (UID: "0942b983-9609-4de3-9896-7bad41758aef"). InnerVolumeSpecName "kube-api-access-4g88h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:22:07.739365 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:07.739344    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g88h\" (UniqueName: \"kubernetes.io/projected/0942b983-9609-4de3-9896-7bad41758aef-kube-api-access-4g88h\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:22:07.739365 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:07.739363    2581 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0942b983-9609-4de3-9896-7bad41758aef-must-gather-output\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Sep 16 18:22:07.765936 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:07.765914    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0942b983-9609-4de3-9896-7bad41758aef" path="/var/lib/kubelet/pods/0942b983-9609-4de3-9896-7bad41758aef/volumes"
Sep 16 18:22:08.000537 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:08.000467    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:22:08.053996 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:08.053974    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlwht_4a459e7e-1c73-4170-b753-22882c81fa6d/konnectivity-agent/0.log" limit=1024
Sep 16 18:22:08.540478 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:08.540462    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sk7zf_must-gather-dlbdm_0942b983-9609-4de3-9896-7bad41758aef/copy/0.log"
Sep 16 18:22:08.540877 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:08.540865    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sk7zf/must-gather-dlbdm"
Sep 16 18:22:08.540988 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:08.540865    2581 scope.go:117] "RemoveContainer" containerID="b9007750462cdd840c2486bef831aaeeb9a3a8d05ad1518cf0c1726108ec5153"
Sep 16 18:22:08.550528 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:08.550511    2581 scope.go:117] "RemoveContainer" containerID="71d53674699bb4e35d1116a91d9d0897674f139e41db5ddc9525e28f74de8887"
Sep 16 18:22:09.903325 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:09.903269    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-975rr_28ee14c3-a8e9-4421-81a1-56f713758016/kube-state-metrics/0.log"
Sep 16 18:22:09.949938 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:09.949918    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-975rr_28ee14c3-a8e9-4421-81a1-56f713758016/kube-rbac-proxy-main/0.log"
Sep 16 18:22:09.992845 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:09.992828    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-975rr_28ee14c3-a8e9-4421-81a1-56f713758016/kube-rbac-proxy-self/0.log"
Sep 16 18:22:10.152272 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.152257    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cw8d_8c9a0a70-06a4-4307-bfa9-4693b933f008/node-exporter/0.log"
Sep 16 18:22:10.187860 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.187816    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cw8d_8c9a0a70-06a4-4307-bfa9-4693b933f008/kube-rbac-proxy/0.log"
Sep 16 18:22:10.226659 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.226642    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cw8d_8c9a0a70-06a4-4307-bfa9-4693b933f008/init-textfile/0.log"
Sep 16 18:22:10.496060 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.496009    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-dmqg7_c451c7c3-281d-421b-b30a-0fb9eab2158b/kube-rbac-proxy-main/0.log"
Sep 16 18:22:10.537775 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.537754    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-dmqg7_c451c7c3-281d-421b-b30a-0fb9eab2158b/kube-rbac-proxy-self/0.log"
Sep 16 18:22:10.571003 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.570987    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-dmqg7_c451c7c3-281d-421b-b30a-0fb9eab2158b/openshift-state-metrics/0.log"
Sep 16 18:22:10.621579 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.621560    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cca209e4-a21d-4f71-9146-c49b09985738/prometheus/0.log"
Sep 16 18:22:10.645982 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.645963    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cca209e4-a21d-4f71-9146-c49b09985738/config-reloader/0.log"
Sep 16 18:22:10.680142 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.680122    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cca209e4-a21d-4f71-9146-c49b09985738/thanos-sidecar/0.log"
Sep 16 18:22:10.714934 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.714917    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cca209e4-a21d-4f71-9146-c49b09985738/kube-rbac-proxy-web/0.log"
Sep 16 18:22:10.748865 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.748822    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cca209e4-a21d-4f71-9146-c49b09985738/kube-rbac-proxy/0.log"
Sep 16 18:22:10.789885 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.789867    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cca209e4-a21d-4f71-9146-c49b09985738/kube-rbac-proxy-thanos/0.log"
Sep 16 18:22:10.836016 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.836000    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cca209e4-a21d-4f71-9146-c49b09985738/init-config-reloader/0.log"
Sep 16 18:22:10.879391 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.879366    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-jwp6h_4c89cf2f-295e-4ac0-9c7b-e4ad9167a496/prometheus-operator/0.log"
Sep 16 18:22:10.911837 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.911817    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-jwp6h_4c89cf2f-295e-4ac0-9c7b-e4ad9167a496/kube-rbac-proxy/0.log"
Sep 16 18:22:10.948014 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:10.947995    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-fcxsr_81d5aa9b-3cae-4ac7-a5fc-d2c1729f14fb/prometheus-operator-admission-webhook/0.log"
Sep 16 18:22:12.451355 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:12.451266    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-fdpj6_68021b3d-3129-4295-a6b4-22454fe05184/networking-console-plugin/0.log"
Sep 16 18:22:12.925965 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:12.925940    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/2.log"
Sep 16 18:22:12.930245 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:12.930223    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xg8vh_9e3a6183-641d-4047-9b87-6b79f0870b33/console-operator/3.log"
Sep 16 18:22:13.666243 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.666218    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"]
Sep 16 18:22:13.666525 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.666505    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0942b983-9609-4de3-9896-7bad41758aef" containerName="gather"
Sep 16 18:22:13.666525 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.666516    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0942b983-9609-4de3-9896-7bad41758aef" containerName="gather"
Sep 16 18:22:13.666592 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.666545    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0942b983-9609-4de3-9896-7bad41758aef" containerName="copy"
Sep 16 18:22:13.666592 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.666550    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0942b983-9609-4de3-9896-7bad41758aef" containerName="copy"
Sep 16 18:22:13.666654 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.666601    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0942b983-9609-4de3-9896-7bad41758aef" containerName="gather"
Sep 16 18:22:13.666654 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.666609    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0942b983-9609-4de3-9896-7bad41758aef" containerName="copy"
Sep 16 18:22:13.671274 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.671258    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.674255 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.674235    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hfg2l\"/\"openshift-service-ca.crt\""
Sep 16 18:22:13.674452 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.674438    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hfg2l\"/\"default-dockercfg-gf7tn\""
Sep 16 18:22:13.675720 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.675690    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hfg2l\"/\"kube-root-ca.crt\""
Sep 16 18:22:13.682108 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.682089    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"]
Sep 16 18:22:13.781686 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.781664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/81684955-051c-4554-a951-4751c61e042b-proc\") pod \"perf-node-gather-daemonset-tw247\" (UID: \"81684955-051c-4554-a951-4751c61e042b\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.781775 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.781693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlvx4\" (UniqueName: \"kubernetes.io/projected/81684955-051c-4554-a951-4751c61e042b-kube-api-access-tlvx4\") pod \"perf-node-gather-daemonset-tw247\" (UID: \"81684955-051c-4554-a951-4751c61e042b\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.781775 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.781759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/81684955-051c-4554-a951-4751c61e042b-podres\") pod \"perf-node-gather-daemonset-tw247\" (UID: \"81684955-051c-4554-a951-4751c61e042b\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.781919 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.781786    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81684955-051c-4554-a951-4751c61e042b-sys\") pod \"perf-node-gather-daemonset-tw247\" (UID: \"81684955-051c-4554-a951-4751c61e042b\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.781919 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.781878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81684955-051c-4554-a951-4751c61e042b-lib-modules\") pod \"perf-node-gather-daemonset-tw247\" (UID: \"81684955-051c-4554-a951-4751c61e042b\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.882451 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.882424    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81684955-051c-4554-a951-4751c61e042b-lib-modules\") pod \"perf-node-gather-daemonset-tw247\" (UID: \"81684955-051c-4554-a951-4751c61e042b\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.882536 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.882467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/81684955-051c-4554-a951-4751c61e042b-proc\") pod \"perf-node-gather-daemonset-tw247\" (UID: \"81684955-051c-4554-a951-4751c61e042b\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.882536 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.882499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlvx4\" (UniqueName: \"kubernetes.io/projected/81684955-051c-4554-a951-4751c61e042b-kube-api-access-tlvx4\") pod \"perf-node-gather-daemonset-tw247\" (UID: \"81684955-051c-4554-a951-4751c61e042b\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.882634 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.882533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/81684955-051c-4554-a951-4751c61e042b-podres\") pod \"perf-node-gather-daemonset-tw247\" (UID: \"81684955-051c-4554-a951-4751c61e042b\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.882634 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.882543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/81684955-051c-4554-a951-4751c61e042b-proc\") pod \"perf-node-gather-daemonset-tw247\" (UID: \"81684955-051c-4554-a951-4751c61e042b\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.882634 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.882559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81684955-051c-4554-a951-4751c61e042b-sys\") pod \"perf-node-gather-daemonset-tw247\" (UID: \"81684955-051c-4554-a951-4751c61e042b\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.882634 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.882584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81684955-051c-4554-a951-4751c61e042b-lib-modules\") pod \"perf-node-gather-daemonset-tw247\" (UID: \"81684955-051c-4554-a951-4751c61e042b\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.882762 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.882676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81684955-051c-4554-a951-4751c61e042b-sys\") pod \"perf-node-gather-daemonset-tw247\" (UID: \"81684955-051c-4554-a951-4751c61e042b\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.882809 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.882788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/81684955-051c-4554-a951-4751c61e042b-podres\") pod \"perf-node-gather-daemonset-tw247\" (UID: \"81684955-051c-4554-a951-4751c61e042b\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.890547 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.890526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlvx4\" (UniqueName: \"kubernetes.io/projected/81684955-051c-4554-a951-4751c61e042b-kube-api-access-tlvx4\") pod \"perf-node-gather-daemonset-tw247\" (UID: \"81684955-051c-4554-a951-4751c61e042b\") " pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:13.980542 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:13.980495    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:14.248776 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:14.248709    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"]
Sep 16 18:22:14.253655 ip-10-0-128-31 kubenswrapper[2581]: W0916 18:22:14.253622    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod81684955_051c_4554_a951_4751c61e042b.slice/crio-b05b49c48cf0f676ddd2611f5ca19f2d93ec6195e4ccfa79beaea2010743f82b WatchSource:0}: Error finding container b05b49c48cf0f676ddd2611f5ca19f2d93ec6195e4ccfa79beaea2010743f82b: Status 404 returned error can't find the container with id b05b49c48cf0f676ddd2611f5ca19f2d93ec6195e4ccfa79beaea2010743f82b
Sep 16 18:22:14.566149 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:14.566124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247" event={"ID":"81684955-051c-4554-a951-4751c61e042b","Type":"ContainerStarted","Data":"61fa28175571fdba2dace6bd070e81be5ac2ae3622eaa01b592e3a522837a2a1"}
Sep 16 18:22:14.566250 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:14.566173    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247" event={"ID":"81684955-051c-4554-a951-4751c61e042b","Type":"ContainerStarted","Data":"b05b49c48cf0f676ddd2611f5ca19f2d93ec6195e4ccfa79beaea2010743f82b"}
Sep 16 18:22:14.566321 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:14.566251    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:14.584262 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:14.584226    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247" podStartSLOduration=1.5842110759999999 podStartE2EDuration="1.584211076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:22:14.583522242 +0000 UTC m=+3463.372144773" watchObservedRunningTime="2026-09-16 18:22:14.584211076 +0000 UTC m=+3463.372833607"
Sep 16 18:22:15.012537 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:15.012480    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-klh2p_a9f7a40a-f3c9-4e73-940e-666151905ce4/dns/0.log"
Sep 16 18:22:15.063617 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:15.063595    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-klh2p_a9f7a40a-f3c9-4e73-940e-666151905ce4/kube-rbac-proxy/0.log"
Sep 16 18:22:15.086939 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:15.086917    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7rg26_22254a42-55b7-4718-abf3-475178a65fc9/dns-node-resolver/0.log"
Sep 16 18:22:15.625286 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:15.625262    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qzm9m_5638697b-24e8-4230-857d-7add1666f79a/node-ca/0.log"
Sep 16 18:22:16.816078 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:16.816050    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zc9kr_d9947ef2-8d1a-4b13-b818-b5b2b7d95760/serve-healthcheck-canary/0.log"
Sep 16 18:22:17.371650 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:17.371628    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n88th_d87112b0-92d4-41bc-985f-572459ec8c6e/kube-rbac-proxy/0.log"
Sep 16 18:22:17.399144 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:17.399119    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n88th_d87112b0-92d4-41bc-985f-572459ec8c6e/exporter/0.log"
Sep 16 18:22:17.423975 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:17.423955    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n88th_d87112b0-92d4-41bc-985f-572459ec8c6e/extractor/0.log"
Sep 16 18:22:20.254215 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:20.254187    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bjxl7_a7f5129f-6c92-41bf-92e4-79da60036a74/s3-init/0.log"
Sep 16 18:22:20.274642 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:20.274622    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bjxl7_a7f5129f-6c92-41bf-92e4-79da60036a74/download-hf-model/0.log"
Sep 16 18:22:20.304363 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:20.304345    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-l4fbb_e7724395-44a9-45f6-bbc3-ac2513b0db86/s3-tls-init-custom/0.log"
Sep 16 18:22:20.325656 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:20.325641    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-l4fbb_e7724395-44a9-45f6-bbc3-ac2513b0db86/download-hf-model/0.log"
Sep 16 18:22:20.363721 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:20.363702    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-x5j5n_cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40/s3-tls-init-serving/0.log"
Sep 16 18:22:20.394661 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:20.394635    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-x5j5n_cfcdbd4e-b7fe-44e3-96fe-17f2e9443e40/download-hf-model/0.log"
Sep 16 18:22:20.577926 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:20.577909    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hfg2l/perf-node-gather-daemonset-tw247"
Sep 16 18:22:24.866383 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:24.866359    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-swmsn_df97691c-0462-40ce-b67c-944b2e45ef2c/migrator/0.log"
Sep 16 18:22:24.891868 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:24.891849    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-swmsn_df97691c-0462-40ce-b67c-944b2e45ef2c/graceful-termination/0.log"
Sep 16 18:22:25.244459 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:25.244440    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-xsmh4_9395826b-e1df-4879-aa43-86e2af895fe8/kube-storage-version-migrator-operator/1.log"
Sep 16 18:22:25.245475 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:25.245458    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-xsmh4_9395826b-e1df-4879-aa43-86e2af895fe8/kube-storage-version-migrator-operator/0.log"
Sep 16 18:22:26.331257 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:26.331227    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nn9tr_1daba9ed-cc2c-4e25-b904-1e2a34af7ce1/kube-multus-additional-cni-plugins/0.log"
Sep 16 18:22:26.350882 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:26.350860    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nn9tr_1daba9ed-cc2c-4e25-b904-1e2a34af7ce1/egress-router-binary-copy/0.log"
Sep 16 18:22:26.371535 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:26.371516    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nn9tr_1daba9ed-cc2c-4e25-b904-1e2a34af7ce1/cni-plugins/0.log"
Sep 16 18:22:26.392017 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:26.391995    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nn9tr_1daba9ed-cc2c-4e25-b904-1e2a34af7ce1/bond-cni-plugin/0.log"
Sep 16 18:22:26.414722 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:26.414704    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nn9tr_1daba9ed-cc2c-4e25-b904-1e2a34af7ce1/routeoverride-cni/0.log"
Sep 16 18:22:26.435372 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:26.435354    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nn9tr_1daba9ed-cc2c-4e25-b904-1e2a34af7ce1/whereabouts-cni-bincopy/0.log"
Sep 16 18:22:26.455926 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:26.455907    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nn9tr_1daba9ed-cc2c-4e25-b904-1e2a34af7ce1/whereabouts-cni/0.log"
Sep 16 18:22:26.658057 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:26.658010    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ncvfq_bdf31547-ab38-4cf0-81d4-c0c02bc3e54e/kube-multus/0.log"
Sep 16 18:22:26.707163 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:26.707144    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jhf5t_fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8/network-metrics-daemon/0.log"
Sep 16 18:22:26.726372 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:26.726356    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jhf5t_fbc6fd3c-7b7f-421f-8318-8d0ef38c4dc8/kube-rbac-proxy/0.log"
Sep 16 18:22:27.577915 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:27.576337    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5kpv5_50a056f9-6c58-4c10-91e6-4c8202e3cc20/ovn-controller/0.log"
Sep 16 18:22:27.595517 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:27.595500    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5kpv5_50a056f9-6c58-4c10-91e6-4c8202e3cc20/ovn-acl-logging/0.log"
Sep 16 18:22:27.623353 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:27.623327    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5kpv5_50a056f9-6c58-4c10-91e6-4c8202e3cc20/kube-rbac-proxy-node/0.log"
Sep 16 18:22:27.650413 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:27.650396    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5kpv5_50a056f9-6c58-4c10-91e6-4c8202e3cc20/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 18:22:27.672309 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:27.672274    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5kpv5_50a056f9-6c58-4c10-91e6-4c8202e3cc20/northd/0.log"
Sep 16 18:22:27.699529 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:27.699509    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5kpv5_50a056f9-6c58-4c10-91e6-4c8202e3cc20/nbdb/0.log"
Sep 16 18:22:27.723672 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:27.723659    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5kpv5_50a056f9-6c58-4c10-91e6-4c8202e3cc20/sbdb/0.log"
Sep 16 18:22:27.768136 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:27.768117    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5kpv5_50a056f9-6c58-4c10-91e6-4c8202e3cc20/ovnkube-controller/0.log"
Sep 16 18:22:29.225319 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:29.225293    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-b7s95_2ef4b019-9c21-40d2-b411-f99d8ed0e41d/check-endpoints/0.log"
Sep 16 18:22:29.308297 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:29.308238    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-ppgtg_bde6ac75-b65f-4477-829e-4e37b5c55b19/network-check-target-container/0.log"
Sep 16 18:22:30.233881 ip-10-0-128-31 kubenswrapper[2581]: I0916 18:22:30.233857    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-749nw_3cd98fb6-5344-470a-a91b-c491ba24c89a/iptables-alerter/0.log"