Sep 17 16:39:43.275133 ip-10-0-135-124 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 16:39:43.275148 ip-10-0-135-124 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 16:39:43.275174 ip-10-0-135-124 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 16:39:43.275474 ip-10-0-135-124 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 16:39:52.057079 ip-10-0-135-124 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 16:39:52.057096 ip-10-0-135-124 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f5723c9535d843e7a108fa71a8745834 --
Sep 17 16:41:21.847400 ip-10-0-135-124 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 16:41:22.290563 ip-10-0-135-124 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:41:22.290563 ip-10-0-135-124 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 16:41:22.290563 ip-10-0-135-124 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:41:22.290563 ip-10-0-135-124 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 16:41:22.290563 ip-10-0-135-124 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:41:22.291587 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.291513    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 16:41:22.298216 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298200    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298217    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298222    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298225    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298228    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298231    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298234    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298238    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298240    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298243    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298246    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298249    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298251    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298254    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298256    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298259    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298262    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298265    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:22.298258 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298268    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298270    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298273    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298283    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298286    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298289    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298291    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298294    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298297    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298301    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298304    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298307    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298310    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298313    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298316    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298319    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298322    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298324    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298327    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:22.298693 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298329    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298332    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298334    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298337    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298340    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298342    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298345    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298347    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298350    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298352    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298355    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298357    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298360    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298363    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298366    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298369    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298372    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298375    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298378    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298380    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:22.299458 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298383    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298385    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298388    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298391    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298393    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298396    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298398    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298400    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298403    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298405    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298408    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298411    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298414    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298416    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298419    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298423    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298426    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298428    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298431    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298433    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298436    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:22.300075 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298439    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298441    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298444    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298446    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298449    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298452    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298455    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298458    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298460    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298463    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298466    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298468    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298471    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298473    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298476    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298479    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298483    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298486    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298488    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:22.300638 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298492    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:22.301286 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.298496    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:22.301557 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301543    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:22.301557 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301556    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301560    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301563    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301566    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301569    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301573    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301576    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301578    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301583    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301587    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301590    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301593    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301595    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301598    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301601    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301603    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301606    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301608    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301611    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:22.301611 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301613    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301616    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301619    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301622    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301625    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301627    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301631    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301635    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301639    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301641    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301644    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301646    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301649    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301651    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301654    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301657    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301659    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301662    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301664    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301667    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:22.302178 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301669    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301672    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301674    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301677    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301679    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301682    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301686    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301689    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301692    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301695    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301698    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301701    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301703    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301706    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301709    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301711    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301714    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301716    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301721    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:22.302705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301725    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301728    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301731    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301733    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301736    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301739    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301742    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301744    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301747    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301750    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301752    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301755    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301757    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301760    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301762    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301765    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301767    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301770    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301773    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301775    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:22.303180 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301778    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301781    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301783    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301786    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301789    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301791    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301794    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301797    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301799    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301802    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301805    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301807    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301810    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301813    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301815    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301818    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301821    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301823    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301826    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:22.303661 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.301828    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302212    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302217    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302221    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302224    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302226    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302229    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302232    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302235    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302237    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302240    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302243    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302247    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302250    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302252    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302255    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302258    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302261    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302263    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:22.304125 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302266    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302269    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302272    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302275    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302278    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302280    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302283    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302285    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302288    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302290    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302293    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302295    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302298    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302300    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302303    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302305    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302308    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302311    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302313    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302316    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:22.304608 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302318    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302321    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302323    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302326    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302329    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302332    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302335    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302338    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302340    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302343    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302345    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302348    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302351    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302354    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302357    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302360    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302362    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302365    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302367    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302370    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:22.305096 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302373    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302377    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302380    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302382    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302384    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302387    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302390    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302392    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302395    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302397    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302399    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302402    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302406    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302410    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302413    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302416    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302418    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302421    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302424    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302426    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:22.305645 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302429    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302432    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302435    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302437    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302440    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302444    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302447    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302450    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302453    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302456    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302458    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302461    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302464    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302468    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302471    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302473    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302476    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302479    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302482    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:22.306130 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302485    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.302487    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302558    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302593    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302599    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302603    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302607    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302610    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302615    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302619    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302622    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302625    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302628    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302631    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302635    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302638    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302641    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302644    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302648    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302651    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302655    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302658    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302661    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302664    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 16:41:22.306599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302668    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302671    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302676    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302678    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302682    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302685    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302688    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302690    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302693    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302696    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302700    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302703    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302706    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302709    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302712    2577 flags.go:64] FLAG: --enable-server="true"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302715    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302720    2577 flags.go:64] FLAG: --event-burst="100"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302723    2577 flags.go:64] FLAG: --event-qps="50"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302726    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302728    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302731    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302735    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302738    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302741    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302744    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 17 16:41:22.307187 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302747    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302750    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302753    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302756    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302759    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302761    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302764    2577 flags.go:64] FLAG: --feature-gates=""
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302768    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302771    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302774    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302777    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302780    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302783    2577 flags.go:64] FLAG: --help="false"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302786    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302789    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302792    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302795    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302798    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302801    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302804    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302807    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302810    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302812    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 16:41:22.307832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302815    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302818    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302821    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302824    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302828    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302831    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302834    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302837    2577 flags.go:64] FLAG: --lock-file=""
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302840    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302843    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302846    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302856    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302859    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302862    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302865    2577 flags.go:64] FLAG: --logging-format="text"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302868    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302871    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302874    2577 flags.go:64] FLAG: --manifest-url=""
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302877    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302881    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302884    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302888    2577 flags.go:64] FLAG: --max-pods="110"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302891    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302894    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302897    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 16:41:22.308449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302900    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302902    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302905    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302908    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302915    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302918    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302921    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302923    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302926    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302931    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302934    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302937    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302942    2577 flags.go:64] FLAG: --port="10250"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302945    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302948    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06382e9bde57a5894"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302951    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302954    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302957    2577 flags.go:64] FLAG: --register-node="true"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302960    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302964    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302967    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302970    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302973    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302976    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302979    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 16:41:22.309069 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302982    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302985    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302988    2577 flags.go:64] FLAG: --runonce="false"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302991    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302994    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.302997    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303000    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303003    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303006    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303009    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303012    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303015    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303018    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303020    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303023    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303026    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303030    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303032    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303037    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303041    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303044    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303048    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303051    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303053    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303056    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 16:41:22.309727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303059    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303062    2577 flags.go:64] FLAG: --v="2"
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303066    2577 flags.go:64] FLAG: --version="false"
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303073    2577 flags.go:64] FLAG: --vmodule=""
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303077    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.303080    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303211    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303215    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303219    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303222    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303225    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303228    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303230    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303234    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303236    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303239    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303242    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303244    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303247    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303249    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303254    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:22.310402 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303257    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303260    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303263    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303266    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303269    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303273    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303275    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303278    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303281    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303284    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303287    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303290    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303293    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303295    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303298    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303301    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303304    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303306    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303309    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303313    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:22.310914 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303316    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303319    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303323    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303325    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303328    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303331    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303334    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303337    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303339    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303342    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303345    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303347    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303350    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303353    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303355    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303358    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303360    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303364    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303367    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303371    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:22.311417 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303375    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303378    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303382    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303384    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303387    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303390    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303392    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303395    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303398    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303401    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303404    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303406    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303409    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303412    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303414    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303417    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303427    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303430    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303433    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303435    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:22.311908 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303438    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303441    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303443    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303446    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303448    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303451    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303454    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303456    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303459    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303463    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303466    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303468    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303471    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303473    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303477    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303480    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303482    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303485    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303488    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:22.312424 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303490    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:22.312931 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303493    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:22.312931 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303496    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:22.312931 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303499    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:22.312931 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.303502    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:22.312931 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.304352    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:41:22.312931 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.311659    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 16:41:22.312931 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.311674    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 16:41:22.312931 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.311694    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 16:41:22.312931 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.311703    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 16:41:22.312931 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311755    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:22.312931 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311760    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:22.312931 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311764    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:22.312931 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311767    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:22.312931 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311770    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:22.312931 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311773    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311776    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311779    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311782    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311785    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311787    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311790    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311793    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311795    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311798    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311801    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311803    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311806    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311808    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311811    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311814    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311817    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311819    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311822    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311825    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:22.313325 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311828    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311831    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311834    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311837    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311840    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311843    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311845    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311848    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311850    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311853    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311856    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311859    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311861    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311864    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311866    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311869    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311871    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311874    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311876    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:22.313801 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311879    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311882    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311884    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311887    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311890    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311892    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311895    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311898    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311900    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311903    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311905    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311908    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311911    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311913    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311916    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311919    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311922    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311925    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311928    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311930    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:22.314270 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311933    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311936    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311938    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311941    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311943    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311946    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311948    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311951    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311953    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311957    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311961    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311965    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311967    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311970    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311974    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311978    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311982    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311985    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311988    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:22.314757 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311991    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:22.315281 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311994    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:22.315281 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.311997    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:22.315281 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312000    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:22.315281 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312003    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:22.315281 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312005    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:22.315281 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312008    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:22.315281 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312011    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:22.315281 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312013    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:22.315281 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312016    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:22.315281 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312019    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:22.315281 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312022    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:22.315281 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312027    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:22.315281 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312030    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:22.315281 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312033    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:22.315281 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312036    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:22.315647 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.312041    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:41:22.315647 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312176    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:22.315647 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312182    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:22.315647 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312185    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:22.315647 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312187    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:22.315647 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312190    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:22.315647 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312193    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:22.315647 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312196    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:22.315647 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312198    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:22.315647 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312201    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:22.315647 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312204    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:22.315647 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312206    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:22.315647 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312209    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:22.315647 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312211    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:22.315647 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312214    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312217    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312220    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312223    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312225    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312229    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312233    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312235    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312238    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312240    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312243    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312246    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312248    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312251    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312253    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312257    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312260    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312262    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312265    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:22.316100 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312267    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312270    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312273    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312275    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312278    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312281    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312283    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312286    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312289    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312293    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312296    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312298    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312301    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312305    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312307    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312310    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312312    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312315    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312317    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:22.316592 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312320    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312322    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312325    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312327    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312330    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312332    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312335    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312337    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312340    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312343    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312346    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312349    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312351    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312354    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312356    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312359    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312361    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312364    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312366    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312369    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:22.317267 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312372    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312374    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312377    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312379    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312382    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312384    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312387    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312390    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312393    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312395    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312398    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312400    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312403    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312406    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312408    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312411    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312414    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312417    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312419    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312422    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:22.317963 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312424    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:22.318520 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312429    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:22.318520 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312432    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:22.318520 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312435    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:22.318520 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312438    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:22.318520 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312441    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:22.318520 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312444    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:22.318520 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:22.312447    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:22.318520 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.312452    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:41:22.318520 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.312539    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 16:41:22.318520 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.314870    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 16:41:22.318520 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.315831    2577 server.go:1019] "Starting client certificate rotation"
Sep 17 16:41:22.318520 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.315944    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 16:41:22.318520 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.315992    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 16:41:22.318917 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.318780    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 16:41:22.320645 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.320627    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 16:41:22.331020 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.331005    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 17 16:41:22.332754 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.332740    2577 log.go:25] "Validated CRI v1 image API"
Sep 17 16:41:22.333566 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.333552    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 16:41:22.337904 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.337884    2577 fs.go:135] Filesystem UUIDs: map[739c10a4-f502-4106-9010-fa9cc7d3b098:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ccac1b13-9f71-4d6e-8ee5-6e68386de20a:/dev/nvme0n1p4]
Sep 17 16:41:22.337968 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.337900    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 16:41:22.341884 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.341784    2577 manager.go:216] Machine: {Timestamp:2026-09-17 16:41:22.341324681 +0000 UTC m=+0.366232961 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104989 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29ae31faebd3f9b1ebda893345f647 SystemUUID:ec29ae31-faeb-d3f9-b1eb-da893345f647 BootID:f5723c95-35d8-43e7-a108-fa71a8745834 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:eb:7d:46:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:eb:7d:46:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:81:8e:0a:a9:57 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 16:41:22.341884 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.341880    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 16:41:22.341989 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.341967    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 16:41:22.344672 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.344644    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 16:41:22.344793 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.344676    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-124.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 16:41:22.345514 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.345505    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 16:41:22.345548 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.345517    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 16:41:22.345548 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.345525    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 16:41:22.345548 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.345539    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 16:41:22.345629 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.345603    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 16:41:22.350303 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.350286    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 16:41:22.351183 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.351171    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 16:41:22.351293 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.351283    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 16:41:22.351328 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.351312    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 16:41:22.351328 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.351321    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 16:41:22.351378 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.351334    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 16:41:22.351378 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.351343    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 17 16:41:22.351378 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.351351    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 16:41:22.351644 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.351631    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 16:41:22.351717 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.351650    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 16:41:22.353967 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.353953    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 16:41:22.356746 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.356733    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 16:41:22.356793 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.356762    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 16:41:22.358091 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.358080    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 16:41:22.358140 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.358095    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 16:41:22.358140 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.358102    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 16:41:22.358140 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.358107    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 16:41:22.358140 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.358117    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 16:41:22.358140 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.358123    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 16:41:22.358140 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.358141    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 16:41:22.358320 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.358146    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 16:41:22.358320 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.358152    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 16:41:22.358320 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.358173    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 16:41:22.358320 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.358192    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 16:41:22.358320 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.358198    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 16:41:22.359149 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.359140    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 16:41:22.359149 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.359149    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 16:41:22.363439 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.363418    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 16:41:22.363652 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.363637    2577 server.go:1268] "Started kubelet"
Sep 17 16:41:22.363747 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.363707    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 16:41:22.363794 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.363776    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 16:41:22.364244 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.364199    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 16:41:22.364244 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.364193    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 16:41:22.364772 ip-10-0-135-124 systemd[1]: Started Kubernetes Kubelet.
Sep 17 16:41:22.365763 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.364819    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 16:41:22.365763 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.365309    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 16:41:22.367308 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.367293    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 16:41:22.369313 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.369280    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 16:41:22.373677 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.372580    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-124.ec2.internal.18d62945f4d62537  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-124.ec2.internal,UID:ip-10-0-135-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-124.ec2.internal,},FirstTimestamp:2026-09-17 16:41:22.363508023 +0000 UTC m=+0.388416295,LastTimestamp:2026-09-17 16:41:22.363508023 +0000 UTC m=+0.388416295,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-124.ec2.internal,}"
Sep 17 16:41:22.374703 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.374689    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 16:41:22.375344 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.375328    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 16:41:22.375523 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.375502    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Sep 17 16:41:22.376195 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.376175    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 16:41:22.376195 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.376185    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 16:41:22.376314 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.376206    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 16:41:22.376314 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.376270    2577 factory.go:55] Registering systemd factory
Sep 17 16:41:22.376314 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.376282    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 16:41:22.376314 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.376292    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 16:41:22.376314 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.376283    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 17 16:41:22.376494 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.376485    2577 factory.go:153] Registering CRI-O factory
Sep 17 16:41:22.376527 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.376496    2577 factory.go:223] Registration of the crio container factory successfully
Sep 17 16:41:22.376555 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.376534    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 16:41:22.376555 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.376547    2577 factory.go:103] Registering Raw factory
Sep 17 16:41:22.376624 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.376557    2577 manager.go:1196] Started watching for new ooms in manager
Sep 17 16:41:22.376820 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.376811    2577 manager.go:349] Starting recovery of all containers
Sep 17 16:41:22.386477 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.386451    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 16:41:22.386741 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.386715    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 16:41:22.387971 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.387952    2577 manager.go:354] Recovery completed
Sep 17 16:41:22.392237 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.392223    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:22.394865 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.394417    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:22.394865 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.394453    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:22.394865 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.394466    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:22.395577 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.395561    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 16:41:22.395577 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.395576    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 16:41:22.395707 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.395593    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 16:41:22.395778 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.395761    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:22.397062 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.397046    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:22.397146 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.397075    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:22.397146 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.397089    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:22.397732 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.397720    2577 policy_none.go:49] "None policy: Start"
Sep 17 16:41:22.397780 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.397735    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 16:41:22.397780 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.397745    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 16:41:22.399693 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.399681    2577 policy_none.go:47] "Start"
Sep 17 16:41:22.401590 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.401469    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h5svm"
Sep 17 16:41:22.406889 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.406470    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-124.ec2.internal.18d62945f6ae2209  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-124.ec2.internal,UID:ip-10-0-135-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-124.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-124.ec2.internal,},FirstTimestamp:2026-09-17 16:41:22.394440201 +0000 UTC m=+0.419348481,LastTimestamp:2026-09-17 16:41:22.394440201 +0000 UTC m=+0.419348481,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-124.ec2.internal,}"
Sep 17 16:41:22.409866 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.409846    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h5svm"
Sep 17 16:41:22.414357 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.414283    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-124.ec2.internal.18d62945f6ae6a9e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-124.ec2.internal,UID:ip-10-0-135-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-124.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-124.ec2.internal,},FirstTimestamp:2026-09-17 16:41:22.394458782 +0000 UTC m=+0.419367062,LastTimestamp:2026-09-17 16:41:22.394458782 +0000 UTC m=+0.419367062,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-124.ec2.internal,}"
Sep 17 16:41:22.424845 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.424832    2577 manager.go:337] "Starting Device Plugin manager"
Sep 17 16:41:22.450408 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.424862    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 16:41:22.450408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.424872    2577 server.go:87] "Starting device plugin registration server"
Sep 17 16:41:22.450408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.425082    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 16:41:22.450408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.425094    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 16:41:22.450408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.425262    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 16:41:22.450408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.425332    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 16:41:22.450408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.425341    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 16:41:22.450408 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.425731    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 16:41:22.450408 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.425775    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-124.ec2.internal\" not found"
Sep 17 16:41:22.450408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.439647    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 16:41:22.450408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.440767    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 16:41:22.450408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.440816    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 16:41:22.450408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.440837    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 16:41:22.450408 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.440900    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 16:41:22.450408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.442782    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:22.525489 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.525453    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:22.526549 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.526534    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:22.526648 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.526565    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:22.526648 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.526578    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:22.526648 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.526606    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.535999 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.535982    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.536083 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.536009    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-124.ec2.internal\": node \"ip-10-0-135-124.ec2.internal\" not found"
Sep 17 16:41:22.541961 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.541891    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal"]
Sep 17 16:41:22.542033 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.541962    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:22.543542 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.543523    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:22.543613 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.543554    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:22.543613 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.543568    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:22.546006 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.545989    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:22.546109 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.546092    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.546178 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.546140    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:22.547711 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.547694    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:22.547810 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.547715    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:22.547810 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.547726    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:22.547810 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.547740    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:22.547810 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.547743    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:22.547810 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.547758    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:22.550070 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.550054    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.550127 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.550079    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:22.550732 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.550721    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:22.550794 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.550740    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:22.550794 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.550749    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:22.555293 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.555277    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Sep 17 16:41:22.578096 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.578077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfdb3c7ac4ecdb67fbea9e7eb6838341-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal\" (UID: \"dfdb3c7ac4ecdb67fbea9e7eb6838341\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.578184 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.578102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c421660fe0335a8468fb794163e0e9e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-124.ec2.internal\" (UID: \"2c421660fe0335a8468fb794163e0e9e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.578184 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.578119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dfdb3c7ac4ecdb67fbea9e7eb6838341-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal\" (UID: \"dfdb3c7ac4ecdb67fbea9e7eb6838341\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.590795 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.590779    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-124.ec2.internal\" not found" node="ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.595134 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.595120    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-124.ec2.internal\" not found" node="ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.655864 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.655846    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Sep 17 16:41:22.678218 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.678196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c421660fe0335a8468fb794163e0e9e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-124.ec2.internal\" (UID: \"2c421660fe0335a8468fb794163e0e9e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.678305 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.678220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dfdb3c7ac4ecdb67fbea9e7eb6838341-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal\" (UID: \"dfdb3c7ac4ecdb67fbea9e7eb6838341\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.678305 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.678240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfdb3c7ac4ecdb67fbea9e7eb6838341-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal\" (UID: \"dfdb3c7ac4ecdb67fbea9e7eb6838341\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.678305 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.678279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfdb3c7ac4ecdb67fbea9e7eb6838341-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal\" (UID: \"dfdb3c7ac4ecdb67fbea9e7eb6838341\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.678305 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.678291    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c421660fe0335a8468fb794163e0e9e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-124.ec2.internal\" (UID: \"2c421660fe0335a8468fb794163e0e9e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.678443 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.678304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dfdb3c7ac4ecdb67fbea9e7eb6838341-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal\" (UID: \"dfdb3c7ac4ecdb67fbea9e7eb6838341\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.756353 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.756323    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Sep 17 16:41:22.857144 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.857068    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Sep 17 16:41:22.895499 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.895477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.900002 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:22.899961    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal"
Sep 17 16:41:22.957869 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:22.957841    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Sep 17 16:41:23.058388 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:23.058362    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Sep 17 16:41:23.158853 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:23.158796    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Sep 17 16:41:23.259381 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:23.259355    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Sep 17 16:41:23.315749 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:23.315728    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 16:41:23.316141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:23.315850    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 16:41:23.335913 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:23.335894    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:23.360201 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:23.360153    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Sep 17 16:41:23.416657 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:23.416588    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 16:36:22 +0000 UTC" deadline="2028-06-11 20:57:35.754206008 +0000 UTC"
Sep 17 16:41:23.416657 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:23.416614    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15196h16m12.337596361s"
Sep 17 16:41:23.457479 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:23.457443    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c421660fe0335a8468fb794163e0e9e.slice/crio-2eed3ec8f7a0cfcee4d18ec6aac48a7633de9c5b04ca4ce60a59601dcc5b6fae WatchSource:0}: Error finding container 2eed3ec8f7a0cfcee4d18ec6aac48a7633de9c5b04ca4ce60a59601dcc5b6fae: Status 404 returned error can't find the container with id 2eed3ec8f7a0cfcee4d18ec6aac48a7633de9c5b04ca4ce60a59601dcc5b6fae
Sep 17 16:41:23.457842 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:23.457821    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfdb3c7ac4ecdb67fbea9e7eb6838341.slice/crio-b303e89f26a73faa3cd453ad189fae60f7ac3754d8c4de807a1a699367154402 WatchSource:0}: Error finding container b303e89f26a73faa3cd453ad189fae60f7ac3754d8c4de807a1a699367154402: Status 404 returned error can't find the container with id b303e89f26a73faa3cd453ad189fae60f7ac3754d8c4de807a1a699367154402
Sep 17 16:41:23.460840 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:23.460815    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Sep 17 16:41:23.462708 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:23.462688    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:41:23.546220 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:23.546198    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:23.576259 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:23.576242    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Sep 17 16:41:23.591089 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:23.591073    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 16:41:23.592136 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:23.592124    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal"
Sep 17 16:41:23.601829 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:23.601816    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 16:41:23.604922 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:23.604910    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:24.352273 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.352241    2577 apiserver.go:52] "Watching apiserver"
Sep 17 16:41:24.361962 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.361932    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 16:41:24.363178 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.363144    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-7dxcc","kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq","openshift-cluster-node-tuning-operator/tuned-brpkg","openshift-dns/node-resolver-fvkfn","openshift-image-registry/node-ca-6d9xb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal","openshift-multus/network-metrics-daemon-fqmmn","kube-system/global-pull-secret-syncer-9snxm","openshift-multus/multus-additional-cni-plugins-phjts","openshift-multus/multus-cv7kh","openshift-network-diagnostics/network-check-target-jn44m","openshift-network-operator/iptables-alerter-l22w8","openshift-ovn-kubernetes/ovnkube-node-2w4sg"]
Sep 17 16:41:24.366815 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.366797    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.369007 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.368988    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l22w8"
Sep 17 16:41:24.369778 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.369757    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 16:41:24.369856 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.369822    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xgdqh\""
Sep 17 16:41:24.369913 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.369757    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 16:41:24.369913 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.369758    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 16:41:24.370093 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.370064    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 16:41:24.370180 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.370145    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 16:41:24.371659 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.371637    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7246b\""
Sep 17 16:41:24.372096 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.371840    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:41:24.372096 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.371866    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.372096 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.371887    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 16:41:24.372096 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.371907    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 16:41:24.373637 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.373561    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.374375 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.374357    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 16:41:24.374459 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.374444    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 16:41:24.374554 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.374542    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4b7zt\""
Sep 17 16:41:24.374715 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.374700    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 16:41:24.375755 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.375740    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 16:41:24.376093 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.376076    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:41:24.376179 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.376114    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 16:41:24.376231 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.376210    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hkw8p\""
Sep 17 16:41:24.376629 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.376612    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fvkfn"
Sep 17 16:41:24.380015 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.379531    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 16:41:24.380015 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.379989    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 16:41:24.380215 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.380194    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hb7x7\""
Sep 17 16:41:24.381016 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.380781    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6d9xb"
Sep 17 16:41:24.383072 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.383057    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:24.383183 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:24.383149    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:24.383305 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.383288    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 16:41:24.383357 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.383302    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 16:41:24.383562 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.383546    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zt6c4\""
Sep 17 16:41:24.383626 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.383612    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 16:41:24.385519 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.385501    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.386882 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.386861    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 16:41:24.387038 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fb52cfe-8eba-442c-8690-e4db91391c47-host-slash\") pod \"iptables-alerter-l22w8\" (UID: \"1fb52cfe-8eba-442c-8690-e4db91391c47\") " pod="openshift-network-operator/iptables-alerter-l22w8"
Sep 17 16:41:24.387108 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-modprobe-d\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.387108 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-systemd\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.387108 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-lib-modules\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.387261 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkjr2\" (UniqueName: \"kubernetes.io/projected/c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1-kube-api-access-tkjr2\") pod \"node-resolver-fvkfn\" (UID: \"c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1\") " pod="openshift-dns/node-resolver-fvkfn"
Sep 17 16:41:24.387261 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-sys\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.387261 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-host\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.387261 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-sysctl-d\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.387420 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1-tmp-dir\") pod \"node-resolver-fvkfn\" (UID: \"c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1\") " pod="openshift-dns/node-resolver-fvkfn"
Sep 17 16:41:24.387420 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b2d9a82-bf68-49e0-b54a-df73639f138a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.387420 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j49z\" (UniqueName: \"kubernetes.io/projected/2b2d9a82-bf68-49e0-b54a-df73639f138a-kube-api-access-7j49z\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.387420 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1fb52cfe-8eba-442c-8690-e4db91391c47-iptables-alerter-script\") pod \"iptables-alerter-l22w8\" (UID: \"1fb52cfe-8eba-442c-8690-e4db91391c47\") " pod="openshift-network-operator/iptables-alerter-l22w8"
Sep 17 16:41:24.387574 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.387574 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-var-lib-kubelet\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.387574 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-sys-fs\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.387574 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-sysconfig\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.387574 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-sysctl-conf\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.387574 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b2d9a82-bf68-49e0-b54a-df73639f138a-cni-binary-copy\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.387574 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtnhj\" (UniqueName: \"kubernetes.io/projected/1fb52cfe-8eba-442c-8690-e4db91391c47-kube-api-access-xtnhj\") pod \"iptables-alerter-l22w8\" (UID: \"1fb52cfe-8eba-442c-8690-e4db91391c47\") " pod="openshift-network-operator/iptables-alerter-l22w8"
Sep 17 16:41:24.387574 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-socket-dir\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-tuned\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndbfv\" (UniqueName: \"kubernetes.io/projected/db7116d5-b098-4a45-8f50-94a2b53e26d2-kube-api-access-ndbfv\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b2d9a82-bf68-49e0-b54a-df73639f138a-cnibin\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-kubernetes\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9bh6\" (UniqueName: \"kubernetes.io/projected/bf2c6a0c-af97-4902-abe7-3a67b751c59c-kube-api-access-q9bh6\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b2d9a82-bf68-49e0-b54a-df73639f138a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-registration-dir\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-device-dir\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db7116d5-b098-4a45-8f50-94a2b53e26d2-tmp\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1-hosts-file\") pod \"node-resolver-fvkfn\" (UID: \"c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1\") " pod="openshift-dns/node-resolver-fvkfn"
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b2d9a82-bf68-49e0-b54a-df73639f138a-system-cni-dir\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-etc-selinux\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-run\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387949    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b2d9a82-bf68-49e0-b54a-df73639f138a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.388011 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.387998    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p9m68\""
Sep 17 16:41:24.388923 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.388145    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 16:41:24.388923 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.388145    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 16:41:24.388923 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.388207    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 16:41:24.388923 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.388253    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 16:41:24.388923 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.388360    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:24.388923 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:24.388427    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:24.389233 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.389020    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 16:41:24.390777 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.390532    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7dxcc"
Sep 17 16:41:24.392756 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.392740    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.392969 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.392953    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f5dkg\""
Sep 17 16:41:24.393077 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.393064    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 16:41:24.393119 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.393074    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 16:41:24.395571 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.395127    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:24.395571 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:24.395210    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:24.395571 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.395314    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2kp5h\""
Sep 17 16:41:24.395571 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.395405    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 16:41:24.410659 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.410630    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fl756"
Sep 17 16:41:24.416804 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.416787    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fl756"
Sep 17 16:41:24.445664 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.445616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal" event={"ID":"2c421660fe0335a8468fb794163e0e9e","Type":"ContainerStarted","Data":"2eed3ec8f7a0cfcee4d18ec6aac48a7633de9c5b04ca4ce60a59601dcc5b6fae"}
Sep 17 16:41:24.447096 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.447070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal" event={"ID":"dfdb3c7ac4ecdb67fbea9e7eb6838341","Type":"ContainerStarted","Data":"b303e89f26a73faa3cd453ad189fae60f7ac3754d8c4de807a1a699367154402"}
Sep 17 16:41:24.477509 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.477491    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 16:41:24.488758 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.488733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9bh6\" (UniqueName: \"kubernetes.io/projected/bf2c6a0c-af97-4902-abe7-3a67b751c59c-kube-api-access-q9bh6\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.488869 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.488767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-registration-dir\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.488869 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.488791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-device-dir\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.488869 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.488816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-var-lib-openvswitch\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.488869 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.488836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-run-ovn\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.488869 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.488850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-cni-bin\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.488869 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.488864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1-hosts-file\") pod \"node-resolver-fvkfn\" (UID: \"c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1\") " pod="openshift-dns/node-resolver-fvkfn"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.488879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b2d9a82-bf68-49e0-b54a-df73639f138a-system-cni-dir\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.488906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-run\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.488930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4c6d895-d5c6-4540-b230-c8d40d3d4205-host\") pod \"node-ca-6d9xb\" (UID: \"f4c6d895-d5c6-4540-b230-c8d40d3d4205\") " pod="openshift-image-registry/node-ca-6d9xb"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.488952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4c6d895-d5c6-4540-b230-c8d40d3d4205-serviceca\") pod \"node-ca-6d9xb\" (UID: \"f4c6d895-d5c6-4540-b230-c8d40d3d4205\") " pod="openshift-image-registry/node-ca-6d9xb"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.488962    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b2d9a82-bf68-49e0-b54a-df73639f138a-system-cni-dir\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.488981    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1-hosts-file\") pod \"node-resolver-fvkfn\" (UID: \"c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1\") " pod="openshift-dns/node-resolver-fvkfn"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.488974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-cni-binary-copy\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.488927    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-device-dir\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-run-multus-certs\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-run-systemd\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-run\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b2d9a82-bf68-49e0-b54a-df73639f138a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fb52cfe-8eba-442c-8690-e4db91391c47-host-slash\") pod \"iptables-alerter-l22w8\" (UID: \"1fb52cfe-8eba-442c-8690-e4db91391c47\") " pod="openshift-network-operator/iptables-alerter-l22w8"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-modprobe-d\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-lib-modules\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.489172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489056    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-registration-dir\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489190    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fb52cfe-8eba-442c-8690-e4db91391c47-host-slash\") pod \"iptables-alerter-l22w8\" (UID: \"1fb52cfe-8eba-442c-8690-e4db91391c47\") " pod="openshift-network-operator/iptables-alerter-l22w8"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-modprobe-d\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l6gj\" (UniqueName: \"kubernetes.io/projected/f4c6d895-d5c6-4540-b230-c8d40d3d4205-kube-api-access-9l6gj\") pod \"node-ca-6d9xb\" (UID: \"f4c6d895-d5c6-4540-b230-c8d40d3d4205\") " pod="openshift-image-registry/node-ca-6d9xb"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd70adfb-9928-407c-b57b-d51617deb1cc-dbus\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b2d9a82-bf68-49e0-b54a-df73639f138a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-lib-modules\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489273    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-multus-socket-dir-parent\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-sys\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-host\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-systemd-units\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-slash\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs\") pod \"network-metrics-daemon-fqmmn\" (UID: \"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a\") " pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a8418cea-e387-4ea4-8a0b-c69dccd07ba5-agent-certs\") pod \"konnectivity-agent-7dxcc\" (UID: \"a8418cea-e387-4ea4-8a0b-c69dccd07ba5\") " pod="kube-system/konnectivity-agent-7dxcc"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-sysctl-d\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1-tmp-dir\") pod \"node-resolver-fvkfn\" (UID: \"c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1\") " pod="openshift-dns/node-resolver-fvkfn"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1fb52cfe-8eba-442c-8690-e4db91391c47-iptables-alerter-script\") pod \"iptables-alerter-l22w8\" (UID: \"1fb52cfe-8eba-442c-8690-e4db91391c47\") " pod="openshift-network-operator/iptables-alerter-l22w8"
Sep 17 16:41:24.489888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-var-lib-kubelet\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489614    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-etc-openvswitch\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-log-socket\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-host\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-sys\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-cni-netd\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489712    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-var-lib-kubelet\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489776    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-sysctl-d\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-system-cni-dir\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-sysconfig\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b2d9a82-bf68-49e0-b54a-df73639f138a-cni-binary-copy\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-sysconfig\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtnhj\" (UniqueName: \"kubernetes.io/projected/1fb52cfe-8eba-442c-8690-e4db91391c47-kube-api-access-xtnhj\") pod \"iptables-alerter-l22w8\" (UID: \"1fb52cfe-8eba-442c-8690-e4db91391c47\") " pod="openshift-network-operator/iptables-alerter-l22w8"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-socket-dir\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489953    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1-tmp-dir\") pod \"node-resolver-fvkfn\" (UID: \"c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1\") " pod="openshift-dns/node-resolver-fvkfn"
Sep 17 16:41:24.490722 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjs5z\" (UniqueName: \"kubernetes.io/projected/4ec628ab-cd9d-4e35-a77c-3ba182d19487-kube-api-access-cjs5z\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.489988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-var-lib-cni-multus\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-multus-conf-dir\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmcxw\" (UniqueName: \"kubernetes.io/projected/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-kube-api-access-tmcxw\") pod \"network-metrics-daemon-fqmmn\" (UID: \"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a\") " pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490064    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-socket-dir\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a8418cea-e387-4ea4-8a0b-c69dccd07ba5-konnectivity-ca\") pod \"konnectivity-agent-7dxcc\" (UID: \"a8418cea-e387-4ea4-8a0b-c69dccd07ba5\") " pod="kube-system/konnectivity-agent-7dxcc"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b2d9a82-bf68-49e0-b54a-df73639f138a-cnibin\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-run-netns\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-hostroot\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b2d9a82-bf68-49e0-b54a-df73639f138a-cnibin\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ec628ab-cd9d-4e35-a77c-3ba182d19487-env-overrides\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b2d9a82-bf68-49e0-b54a-df73639f138a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1fb52cfe-8eba-442c-8690-e4db91391c47-iptables-alerter-script\") pod \"iptables-alerter-l22w8\" (UID: \"1fb52cfe-8eba-442c-8690-e4db91391c47\") " pod="openshift-network-operator/iptables-alerter-l22w8"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db7116d5-b098-4a45-8f50-94a2b53e26d2-tmp\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-etc-kubernetes\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt2xv\" (UniqueName: \"kubernetes.io/projected/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-kube-api-access-nt2xv\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.491544 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-run-openvswitch\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490416    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b2d9a82-bf68-49e0-b54a-df73639f138a-cni-binary-copy\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-node-log\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58fzx\" (UniqueName: \"kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx\") pod \"network-check-target-jn44m\" (UID: \"2c9aee2e-6b4d-4c04-9a63-d2f81312f45f\") " pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-etc-selinux\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-var-lib-cni-bin\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-systemd\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490601    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-etc-selinux\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-multus-cni-dir\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-cnibin\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b2d9a82-bf68-49e0-b54a-df73639f138a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-systemd\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-multus-daemon-config\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkjr2\" (UniqueName: \"kubernetes.io/projected/c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1-kube-api-access-tkjr2\") pod \"node-resolver-fvkfn\" (UID: \"c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1\") " pod="openshift-dns/node-resolver-fvkfn"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490734    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-run-ovn-kubernetes\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.492214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-var-lib-kubelet\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b2d9a82-bf68-49e0-b54a-df73639f138a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j49z\" (UniqueName: \"kubernetes.io/projected/2b2d9a82-bf68-49e0-b54a-df73639f138a-kube-api-access-7j49z\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-run-k8s-cni-cncf-io\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-kubelet\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ec628ab-cd9d-4e35-a77c-3ba182d19487-ovnkube-config\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ec628ab-cd9d-4e35-a77c-3ba182d19487-ovnkube-script-lib\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-tuned\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.490998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-sys-fs\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.491027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-sysctl-conf\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.491087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf2c6a0c-af97-4902-abe7-3a67b751c59c-sys-fs\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.491084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ec628ab-cd9d-4e35-a77c-3ba182d19487-ovn-node-metrics-cert\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.491179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndbfv\" (UniqueName: \"kubernetes.io/projected/db7116d5-b098-4a45-8f50-94a2b53e26d2-kube-api-access-ndbfv\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.491210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd70adfb-9928-407c-b57b-d51617deb1cc-kubelet-config\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.491232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-sysctl-conf\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.491249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-run-netns\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.491295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-kubernetes\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.492847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.491366    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b2d9a82-bf68-49e0-b54a-df73639f138a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.493515 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.491394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-kubernetes\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.494473 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.494448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/db7116d5-b098-4a45-8f50-94a2b53e26d2-etc-tuned\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.494473 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.494466    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db7116d5-b098-4a45-8f50-94a2b53e26d2-tmp\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.498198 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.498140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9bh6\" (UniqueName: \"kubernetes.io/projected/bf2c6a0c-af97-4902-abe7-3a67b751c59c-kube-api-access-q9bh6\") pod \"aws-ebs-csi-driver-node-k7xrq\" (UID: \"bf2c6a0c-af97-4902-abe7-3a67b751c59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.498549 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.498526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j49z\" (UniqueName: \"kubernetes.io/projected/2b2d9a82-bf68-49e0-b54a-df73639f138a-kube-api-access-7j49z\") pod \"multus-additional-cni-plugins-phjts\" (UID: \"2b2d9a82-bf68-49e0-b54a-df73639f138a\") " pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.498810 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.498785    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtnhj\" (UniqueName: \"kubernetes.io/projected/1fb52cfe-8eba-442c-8690-e4db91391c47-kube-api-access-xtnhj\") pod \"iptables-alerter-l22w8\" (UID: \"1fb52cfe-8eba-442c-8690-e4db91391c47\") " pod="openshift-network-operator/iptables-alerter-l22w8"
Sep 17 16:41:24.499020 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.498992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkjr2\" (UniqueName: \"kubernetes.io/projected/c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1-kube-api-access-tkjr2\") pod \"node-resolver-fvkfn\" (UID: \"c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1\") " pod="openshift-dns/node-resolver-fvkfn"
Sep 17 16:41:24.499096 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.499025    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndbfv\" (UniqueName: \"kubernetes.io/projected/db7116d5-b098-4a45-8f50-94a2b53e26d2-kube-api-access-ndbfv\") pod \"tuned-brpkg\" (UID: \"db7116d5-b098-4a45-8f50-94a2b53e26d2\") " pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.591787 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.591757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-run-ovn-kubernetes\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592046 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.591796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-var-lib-kubelet\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.592046 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.591817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-run-k8s-cni-cncf-io\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.592046 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.591840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-kubelet\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592046 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.591862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ec628ab-cd9d-4e35-a77c-3ba182d19487-ovnkube-config\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592046 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.591876    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-run-ovn-kubernetes\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592046 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.591884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ec628ab-cd9d-4e35-a77c-3ba182d19487-ovnkube-script-lib\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592046 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.591919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ec628ab-cd9d-4e35-a77c-3ba182d19487-ovn-node-metrics-cert\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592046 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.591943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd70adfb-9928-407c-b57b-d51617deb1cc-kubelet-config\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:24.592046 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.591965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-run-netns\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592046 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.591992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-var-lib-openvswitch\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592046 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-run-ovn\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592046 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-cni-bin\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4c6d895-d5c6-4540-b230-c8d40d3d4205-host\") pod \"node-ca-6d9xb\" (UID: \"f4c6d895-d5c6-4540-b230-c8d40d3d4205\") " pod="openshift-image-registry/node-ca-6d9xb"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4c6d895-d5c6-4540-b230-c8d40d3d4205-serviceca\") pod \"node-ca-6d9xb\" (UID: \"f4c6d895-d5c6-4540-b230-c8d40d3d4205\") " pod="openshift-image-registry/node-ca-6d9xb"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-cni-binary-copy\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-run-multus-certs\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-run-systemd\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l6gj\" (UniqueName: \"kubernetes.io/projected/f4c6d895-d5c6-4540-b230-c8d40d3d4205-kube-api-access-9l6gj\") pod \"node-ca-6d9xb\" (UID: \"f4c6d895-d5c6-4540-b230-c8d40d3d4205\") " pod="openshift-image-registry/node-ca-6d9xb"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd70adfb-9928-407c-b57b-d51617deb1cc-dbus\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-multus-socket-dir-parent\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-systemd-units\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-slash\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs\") pod \"network-metrics-daemon-fqmmn\" (UID: \"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a\") " pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a8418cea-e387-4ea4-8a0b-c69dccd07ba5-agent-certs\") pod \"konnectivity-agent-7dxcc\" (UID: \"a8418cea-e387-4ea4-8a0b-c69dccd07ba5\") " pod="kube-system/konnectivity-agent-7dxcc"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-etc-openvswitch\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-log-socket\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-cni-netd\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-system-cni-dir\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ec628ab-cd9d-4e35-a77c-3ba182d19487-ovnkube-script-lib\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.592750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjs5z\" (UniqueName: \"kubernetes.io/projected/4ec628ab-cd9d-4e35-a77c-3ba182d19487-kube-api-access-cjs5z\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-var-lib-kubelet\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-var-lib-cni-multus\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-multus-conf-dir\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmcxw\" (UniqueName: \"kubernetes.io/projected/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-kube-api-access-tmcxw\") pod \"network-metrics-daemon-fqmmn\" (UID: \"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a\") " pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592600    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd70adfb-9928-407c-b57b-d51617deb1cc-kubelet-config\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a8418cea-e387-4ea4-8a0b-c69dccd07ba5-konnectivity-ca\") pod \"konnectivity-agent-7dxcc\" (UID: \"a8418cea-e387-4ea4-8a0b-c69dccd07ba5\") " pod="kube-system/konnectivity-agent-7dxcc"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-run-netns\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592637    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-run-netns\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-hostroot\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-run-netns\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ec628ab-cd9d-4e35-a77c-3ba182d19487-env-overrides\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-run-k8s-cni-cncf-io\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-etc-kubernetes\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt2xv\" (UniqueName: \"kubernetes.io/projected/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-kube-api-access-nt2xv\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592750    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-kubelet\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-run-openvswitch\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.593478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-node-log\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fzx\" (UniqueName: \"kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx\") pod \"network-check-target-jn44m\" (UID: \"2c9aee2e-6b4d-4c04-9a63-d2f81312f45f\") " pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-var-lib-cni-bin\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-multus-cni-dir\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-cnibin\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-multus-daemon-config\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593031    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-var-lib-openvswitch\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-run-ovn\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593148    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-cni-bin\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ec628ab-cd9d-4e35-a77c-3ba182d19487-ovnkube-config\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.592553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-var-lib-cni-multus\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4c6d895-d5c6-4540-b230-c8d40d3d4205-host\") pod \"node-ca-6d9xb\" (UID: \"f4c6d895-d5c6-4540-b230-c8d40d3d4205\") " pod="openshift-image-registry/node-ca-6d9xb"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-systemd-units\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-cni-netd\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593275    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-system-cni-dir\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd70adfb-9928-407c-b57b-d51617deb1cc-dbus\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:24.594444 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:24.593388    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:24.593457    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs podName:35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a nodeName:}" failed. No retries permitted until 2026-09-17 16:41:25.093437946 +0000 UTC m=+3.118346219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs") pod "network-metrics-daemon-fqmmn" (UID: "35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593687    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-log-socket\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4c6d895-d5c6-4540-b230-c8d40d3d4205-serviceca\") pod \"node-ca-6d9xb\" (UID: \"f4c6d895-d5c6-4540-b230-c8d40d3d4205\") " pod="openshift-image-registry/node-ca-6d9xb"
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593771    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a8418cea-e387-4ea4-8a0b-c69dccd07ba5-konnectivity-ca\") pod \"konnectivity-agent-7dxcc\" (UID: \"a8418cea-e387-4ea4-8a0b-c69dccd07ba5\") " pod="kube-system/konnectivity-agent-7dxcc"
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-etc-openvswitch\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ec628ab-cd9d-4e35-a77c-3ba182d19487-env-overrides\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-multus-cni-dir\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-run-systemd\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593945    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-run-multus-certs\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593953    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-multus-conf-dir\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.594000    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-cnibin\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.593999    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-multus-socket-dir-parent\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.594033    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-host-var-lib-cni-bin\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.594059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-etc-kubernetes\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:24.594078    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.594098    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-node-log\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.595141 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:24.594119    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret podName:fd70adfb-9928-407c-b57b-d51617deb1cc nodeName:}" failed. No retries permitted until 2026-09-17 16:41:25.09410578 +0000 UTC m=+3.119014046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret") pod "global-pull-secret-syncer-9snxm" (UID: "fd70adfb-9928-407c-b57b-d51617deb1cc") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:24.595967 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.594138    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-host-slash\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.595967 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.594182    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec628ab-cd9d-4e35-a77c-3ba182d19487-run-openvswitch\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.595967 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.594235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-hostroot\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.595967 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.594270    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-cni-binary-copy\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.595967 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.594380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-multus-daemon-config\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.596433 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.596412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ec628ab-cd9d-4e35-a77c-3ba182d19487-ovn-node-metrics-cert\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.596938 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.596916    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a8418cea-e387-4ea4-8a0b-c69dccd07ba5-agent-certs\") pod \"konnectivity-agent-7dxcc\" (UID: \"a8418cea-e387-4ea4-8a0b-c69dccd07ba5\") " pod="kube-system/konnectivity-agent-7dxcc"
Sep 17 16:41:24.601755 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:24.601728    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:24.601755 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:24.601751    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:24.601902 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:24.601763    2577 projected.go:196] Error preparing data for projected volume kube-api-access-58fzx for pod openshift-network-diagnostics/network-check-target-jn44m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:24.601902 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:24.601812    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx podName:2c9aee2e-6b4d-4c04-9a63-d2f81312f45f nodeName:}" failed. No retries permitted until 2026-09-17 16:41:25.101798263 +0000 UTC m=+3.126706552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-58fzx" (UniqueName: "kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx") pod "network-check-target-jn44m" (UID: "2c9aee2e-6b4d-4c04-9a63-d2f81312f45f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:24.603467 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.603413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt2xv\" (UniqueName: \"kubernetes.io/projected/9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe-kube-api-access-nt2xv\") pod \"multus-cv7kh\" (UID: \"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe\") " pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.603467 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.603429    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjs5z\" (UniqueName: \"kubernetes.io/projected/4ec628ab-cd9d-4e35-a77c-3ba182d19487-kube-api-access-cjs5z\") pod \"ovnkube-node-2w4sg\" (UID: \"4ec628ab-cd9d-4e35-a77c-3ba182d19487\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.604123 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.604104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l6gj\" (UniqueName: \"kubernetes.io/projected/f4c6d895-d5c6-4540-b230-c8d40d3d4205-kube-api-access-9l6gj\") pod \"node-ca-6d9xb\" (UID: \"f4c6d895-d5c6-4540-b230-c8d40d3d4205\") " pod="openshift-image-registry/node-ca-6d9xb"
Sep 17 16:41:24.606374 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.606332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmcxw\" (UniqueName: \"kubernetes.io/projected/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-kube-api-access-tmcxw\") pod \"network-metrics-daemon-fqmmn\" (UID: \"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a\") " pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:24.677920 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.677895    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-phjts"
Sep 17 16:41:24.684543 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.684522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l22w8"
Sep 17 16:41:24.694104 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.694083    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq"
Sep 17 16:41:24.699724 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.699698    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-brpkg"
Sep 17 16:41:24.708228 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.708211    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fvkfn"
Sep 17 16:41:24.714835 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.714680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6d9xb"
Sep 17 16:41:24.722363 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.722345    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:24.733366 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.733349    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7dxcc"
Sep 17 16:41:24.738962 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.738946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cv7kh"
Sep 17 16:41:24.874888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:24.874815    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:25.097313 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:25.097284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs\") pod \"network-metrics-daemon-fqmmn\" (UID: \"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a\") " pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:25.097470 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:25.097324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:25.097470 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:25.097451    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:25.097541 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:25.097509    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret podName:fd70adfb-9928-407c-b57b-d51617deb1cc nodeName:}" failed. No retries permitted until 2026-09-17 16:41:26.097496049 +0000 UTC m=+4.122404316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret") pod "global-pull-secret-syncer-9snxm" (UID: "fd70adfb-9928-407c-b57b-d51617deb1cc") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:25.097583 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:25.097452    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:25.097625 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:25.097613    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs podName:35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a nodeName:}" failed. No retries permitted until 2026-09-17 16:41:26.097595485 +0000 UTC m=+4.122503774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs") pod "network-metrics-daemon-fqmmn" (UID: "35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:25.198546 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:25.198469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fzx\" (UniqueName: \"kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx\") pod \"network-check-target-jn44m\" (UID: \"2c9aee2e-6b4d-4c04-9a63-d2f81312f45f\") " pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:25.198670 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:25.198622    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:25.198670 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:25.198639    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:25.198670 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:25.198652    2577 projected.go:196] Error preparing data for projected volume kube-api-access-58fzx for pod openshift-network-diagnostics/network-check-target-jn44m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:25.198790 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:25.198708    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx podName:2c9aee2e-6b4d-4c04-9a63-d2f81312f45f nodeName:}" failed. No retries permitted until 2026-09-17 16:41:26.198692496 +0000 UTC m=+4.223600776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-58fzx" (UniqueName: "kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx") pod "network-check-target-jn44m" (UID: "2c9aee2e-6b4d-4c04-9a63-d2f81312f45f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:25.420002 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:25.419921    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 16:36:24 +0000 UTC" deadline="2028-04-09 14:43:05.390834731 +0000 UTC"
Sep 17 16:41:25.420002 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:25.419967    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13678h1m39.970871614s"
Sep 17 16:41:25.420439 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:25.420416    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c6d895_d5c6_4540_b230_c8d40d3d4205.slice/crio-0aa5c8fb398e7a9d643592bf39665d9eadaf6ef1f950f9f1917ffb7b76ec8763 WatchSource:0}: Error finding container 0aa5c8fb398e7a9d643592bf39665d9eadaf6ef1f950f9f1917ffb7b76ec8763: Status 404 returned error can't find the container with id 0aa5c8fb398e7a9d643592bf39665d9eadaf6ef1f950f9f1917ffb7b76ec8763
Sep 17 16:41:25.422071 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:25.422049    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb52cfe_8eba_442c_8690_e4db91391c47.slice/crio-af47fb1c2fa43c5ab3eef4c19d173d4f8348c2592c88cbe487f98d150ac08c80 WatchSource:0}: Error finding container af47fb1c2fa43c5ab3eef4c19d173d4f8348c2592c88cbe487f98d150ac08c80: Status 404 returned error can't find the container with id af47fb1c2fa43c5ab3eef4c19d173d4f8348c2592c88cbe487f98d150ac08c80
Sep 17 16:41:25.423829 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:25.423792    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec628ab_cd9d_4e35_a77c_3ba182d19487.slice/crio-529aa7f8f662ddff4763e50343bd658de6f639f037903e3544b268eed1217cca WatchSource:0}: Error finding container 529aa7f8f662ddff4763e50343bd658de6f639f037903e3544b268eed1217cca: Status 404 returned error can't find the container with id 529aa7f8f662ddff4763e50343bd658de6f639f037903e3544b268eed1217cca
Sep 17 16:41:25.427331 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:25.427308    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2c6a0c_af97_4902_abe7_3a67b751c59c.slice/crio-b66c3105af9f0eec217f924fe6cc091201d762fcc316050b8bc3885336507bb8 WatchSource:0}: Error finding container b66c3105af9f0eec217f924fe6cc091201d762fcc316050b8bc3885336507bb8: Status 404 returned error can't find the container with id b66c3105af9f0eec217f924fe6cc091201d762fcc316050b8bc3885336507bb8
Sep 17 16:41:25.427745 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:25.427662    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2d9a82_bf68_49e0_b54a_df73639f138a.slice/crio-ff821e33511dc12b0c6a32654126c399da9287461f99a3ff3caafa9d3570be5b WatchSource:0}: Error finding container ff821e33511dc12b0c6a32654126c399da9287461f99a3ff3caafa9d3570be5b: Status 404 returned error can't find the container with id ff821e33511dc12b0c6a32654126c399da9287461f99a3ff3caafa9d3570be5b
Sep 17 16:41:25.428818 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:25.428793    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8418cea_e387_4ea4_8a0b_c69dccd07ba5.slice/crio-71aef1c721b60a69fdfa7e3522436307c2ca65c2907f32010c61721b40e3a4da WatchSource:0}: Error finding container 71aef1c721b60a69fdfa7e3522436307c2ca65c2907f32010c61721b40e3a4da: Status 404 returned error can't find the container with id 71aef1c721b60a69fdfa7e3522436307c2ca65c2907f32010c61721b40e3a4da
Sep 17 16:41:25.450176 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:25.449968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fvkfn" event={"ID":"c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1","Type":"ContainerStarted","Data":"9b82cdb4e22de070bace4759b261e0b565c500db7f935291b3b59dd5f674680f"}
Sep 17 16:41:25.451684 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:25.451663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7dxcc" event={"ID":"a8418cea-e387-4ea4-8a0b-c69dccd07ba5","Type":"ContainerStarted","Data":"71aef1c721b60a69fdfa7e3522436307c2ca65c2907f32010c61721b40e3a4da"}
Sep 17 16:41:25.452696 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:25.452677    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg" event={"ID":"4ec628ab-cd9d-4e35-a77c-3ba182d19487","Type":"ContainerStarted","Data":"529aa7f8f662ddff4763e50343bd658de6f639f037903e3544b268eed1217cca"}
Sep 17 16:41:25.453780 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:25.453746    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6d9xb" event={"ID":"f4c6d895-d5c6-4540-b230-c8d40d3d4205","Type":"ContainerStarted","Data":"0aa5c8fb398e7a9d643592bf39665d9eadaf6ef1f950f9f1917ffb7b76ec8763"}
Sep 17 16:41:25.454982 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:25.454957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cv7kh" event={"ID":"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe","Type":"ContainerStarted","Data":"f70f21de22ec3ed0398be764dab8a22c12f85c9f7aef5eca2dfce7d81f00727b"}
Sep 17 16:41:25.456110 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:25.456090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-brpkg" event={"ID":"db7116d5-b098-4a45-8f50-94a2b53e26d2","Type":"ContainerStarted","Data":"e58592fd50f07cb623c3bcb28379a9d919802956997331ad4bcfaeb7c5c44aff"}
Sep 17 16:41:25.457930 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:25.457907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phjts" event={"ID":"2b2d9a82-bf68-49e0-b54a-df73639f138a","Type":"ContainerStarted","Data":"ff821e33511dc12b0c6a32654126c399da9287461f99a3ff3caafa9d3570be5b"}
Sep 17 16:41:25.459006 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:25.458975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq" event={"ID":"bf2c6a0c-af97-4902-abe7-3a67b751c59c","Type":"ContainerStarted","Data":"b66c3105af9f0eec217f924fe6cc091201d762fcc316050b8bc3885336507bb8"}
Sep 17 16:41:25.461851 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:25.461828    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l22w8" event={"ID":"1fb52cfe-8eba-442c-8690-e4db91391c47","Type":"ContainerStarted","Data":"af47fb1c2fa43c5ab3eef4c19d173d4f8348c2592c88cbe487f98d150ac08c80"}
Sep 17 16:41:26.104871 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:26.104786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs\") pod \"network-metrics-daemon-fqmmn\" (UID: \"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a\") " pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:26.104871 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:26.104844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:26.105085 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:26.104977    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:26.105085 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:26.105035    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret podName:fd70adfb-9928-407c-b57b-d51617deb1cc nodeName:}" failed. No retries permitted until 2026-09-17 16:41:28.105017159 +0000 UTC m=+6.129925432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret") pod "global-pull-secret-syncer-9snxm" (UID: "fd70adfb-9928-407c-b57b-d51617deb1cc") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:26.105444 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:26.105425    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:26.105528 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:26.105482    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs podName:35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a nodeName:}" failed. No retries permitted until 2026-09-17 16:41:28.105466722 +0000 UTC m=+6.130375001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs") pod "network-metrics-daemon-fqmmn" (UID: "35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:26.205689 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:26.205642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fzx\" (UniqueName: \"kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx\") pod \"network-check-target-jn44m\" (UID: \"2c9aee2e-6b4d-4c04-9a63-d2f81312f45f\") " pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:26.205866 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:26.205851    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:26.205923 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:26.205872    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:26.205923 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:26.205885    2577 projected.go:196] Error preparing data for projected volume kube-api-access-58fzx for pod openshift-network-diagnostics/network-check-target-jn44m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:26.206039 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:26.205941    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx podName:2c9aee2e-6b4d-4c04-9a63-d2f81312f45f nodeName:}" failed. No retries permitted until 2026-09-17 16:41:28.205922227 +0000 UTC m=+6.230830499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-58fzx" (UniqueName: "kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx") pod "network-check-target-jn44m" (UID: "2c9aee2e-6b4d-4c04-9a63-d2f81312f45f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:26.420675 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:26.420585    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 16:36:24 +0000 UTC" deadline="2028-04-24 18:12:29.734212826 +0000 UTC"
Sep 17 16:41:26.420675 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:26.420629    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14041h31m3.313588817s"
Sep 17 16:41:26.452767 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:26.449320    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:26.452767 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:26.449493    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:26.452767 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:26.449513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:26.452767 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:26.449587    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:26.452767 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:26.449930    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:26.452767 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:26.450497    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:26.477853 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:26.477198    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal" event={"ID":"2c421660fe0335a8468fb794163e0e9e","Type":"ContainerStarted","Data":"901856f3ab9675d8f098da3e864dde0cb9ca34a2a0cf2f7da04cb06262e2836d"}
Sep 17 16:41:26.486105 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:26.485511    2577 generic.go:356] "Generic (PLEG): container finished" podID="dfdb3c7ac4ecdb67fbea9e7eb6838341" containerID="cfa7de6967703ae8432cb61a76b91c52bb91596ce2fe659c1add2b3406861071" exitCode=0
Sep 17 16:41:26.486105 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:26.485550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal" event={"ID":"dfdb3c7ac4ecdb67fbea9e7eb6838341","Type":"ContainerDied","Data":"cfa7de6967703ae8432cb61a76b91c52bb91596ce2fe659c1add2b3406861071"}
Sep 17 16:41:26.496752 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:26.496235    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal" podStartSLOduration=3.4962167219999998 podStartE2EDuration="3.496216722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:26.495442063 +0000 UTC m=+4.520350352" watchObservedRunningTime="2026-09-17 16:41:26.496216722 +0000 UTC m=+4.521125011"
Sep 17 16:41:27.500080 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:27.500047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal" event={"ID":"dfdb3c7ac4ecdb67fbea9e7eb6838341","Type":"ContainerStarted","Data":"425235dafce4f93882eacd241ab713cda81ad5600f24e7e37222def483b2773e"}
Sep 17 16:41:27.515448 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:27.515397    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal" podStartSLOduration=4.515376919 podStartE2EDuration="4.515376919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:27.514683863 +0000 UTC m=+5.539592152" watchObservedRunningTime="2026-09-17 16:41:27.515376919 +0000 UTC m=+5.540285211"
Sep 17 16:41:28.121606 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:28.121570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs\") pod \"network-metrics-daemon-fqmmn\" (UID: \"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a\") " pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:28.121774 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:28.121629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:28.121853 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:28.121812    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:28.121905 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:28.121879    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret podName:fd70adfb-9928-407c-b57b-d51617deb1cc nodeName:}" failed. No retries permitted until 2026-09-17 16:41:32.121860279 +0000 UTC m=+10.146768550 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret") pod "global-pull-secret-syncer-9snxm" (UID: "fd70adfb-9928-407c-b57b-d51617deb1cc") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:28.122378 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:28.122331    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:28.122489 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:28.122381    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs podName:35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a nodeName:}" failed. No retries permitted until 2026-09-17 16:41:32.12236646 +0000 UTC m=+10.147274740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs") pod "network-metrics-daemon-fqmmn" (UID: "35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:28.222054 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:28.222017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fzx\" (UniqueName: \"kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx\") pod \"network-check-target-jn44m\" (UID: \"2c9aee2e-6b4d-4c04-9a63-d2f81312f45f\") " pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:28.222262 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:28.222190    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:28.222262 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:28.222210    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:28.222262 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:28.222223    2577 projected.go:196] Error preparing data for projected volume kube-api-access-58fzx for pod openshift-network-diagnostics/network-check-target-jn44m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:28.222433 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:28.222279    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx podName:2c9aee2e-6b4d-4c04-9a63-d2f81312f45f nodeName:}" failed. No retries permitted until 2026-09-17 16:41:32.222260958 +0000 UTC m=+10.247169228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-58fzx" (UniqueName: "kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx") pod "network-check-target-jn44m" (UID: "2c9aee2e-6b4d-4c04-9a63-d2f81312f45f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:28.444414 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:28.444339    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:28.444629 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:28.444463    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:28.444861 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:28.444843    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:28.444959 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:28.444941    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:28.445024 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:28.444992    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:28.445080 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:28.445068    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:30.441759 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:30.441724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:30.442218 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:30.441724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:30.442218 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:30.441967    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:30.442218 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:30.441863    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:30.442218 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:30.442015    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:30.442218 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:30.442087    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:32.153656 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:32.153618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs\") pod \"network-metrics-daemon-fqmmn\" (UID: \"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a\") " pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:32.154090 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:32.153674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:32.154090 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:32.153771    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:32.154090 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:32.153833    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs podName:35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a nodeName:}" failed. No retries permitted until 2026-09-17 16:41:40.153810939 +0000 UTC m=+18.178719207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs") pod "network-metrics-daemon-fqmmn" (UID: "35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:32.154090 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:32.153778    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:32.154090 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:32.153898    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret podName:fd70adfb-9928-407c-b57b-d51617deb1cc nodeName:}" failed. No retries permitted until 2026-09-17 16:41:40.15387987 +0000 UTC m=+18.178788138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret") pod "global-pull-secret-syncer-9snxm" (UID: "fd70adfb-9928-407c-b57b-d51617deb1cc") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:32.254982 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:32.254943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fzx\" (UniqueName: \"kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx\") pod \"network-check-target-jn44m\" (UID: \"2c9aee2e-6b4d-4c04-9a63-d2f81312f45f\") " pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:32.255140 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:32.255111    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:32.255140 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:32.255127    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:32.255140 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:32.255140    2577 projected.go:196] Error preparing data for projected volume kube-api-access-58fzx for pod openshift-network-diagnostics/network-check-target-jn44m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:32.255295 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:32.255204    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx podName:2c9aee2e-6b4d-4c04-9a63-d2f81312f45f nodeName:}" failed. No retries permitted until 2026-09-17 16:41:40.255186996 +0000 UTC m=+18.280095266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-58fzx" (UniqueName: "kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx") pod "network-check-target-jn44m" (UID: "2c9aee2e-6b4d-4c04-9a63-d2f81312f45f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:32.442317 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:32.442244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:32.442468 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:32.442352    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:32.442468 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:32.442421    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:32.442588 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:32.442499    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:32.442648 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:32.442586    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:32.442847 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:32.442770    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:34.441476 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:34.441442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:34.441865 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:34.441442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:34.441865 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:34.441571    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:34.441865 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:34.441450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:34.441865 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:34.441614    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:34.441865 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:34.441710    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:36.441388 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:36.441352    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:36.441820 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:36.441352    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:36.441820 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:36.441478    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:36.441820 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:36.441576    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:36.441820 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:36.441357    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:36.441820 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:36.441701    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:38.441503 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:38.441473    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:38.441930 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:38.441479    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:38.441930 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:38.441606    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:38.441930 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:38.441634    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:38.441930 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:38.441678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:38.441930 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:38.441776    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:40.213731 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:40.213694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:40.214274 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:40.213797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs\") pod \"network-metrics-daemon-fqmmn\" (UID: \"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a\") " pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:40.214274 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:40.213857    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:40.214274 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:40.213897    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:40.214274 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:40.213937    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret podName:fd70adfb-9928-407c-b57b-d51617deb1cc nodeName:}" failed. No retries permitted until 2026-09-17 16:41:56.213916133 +0000 UTC m=+34.238824415 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret") pod "global-pull-secret-syncer-9snxm" (UID: "fd70adfb-9928-407c-b57b-d51617deb1cc") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:40.214274 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:40.213957    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs podName:35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a nodeName:}" failed. No retries permitted until 2026-09-17 16:41:56.213946852 +0000 UTC m=+34.238855122 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs") pod "network-metrics-daemon-fqmmn" (UID: "35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:40.314949 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:40.314914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fzx\" (UniqueName: \"kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx\") pod \"network-check-target-jn44m\" (UID: \"2c9aee2e-6b4d-4c04-9a63-d2f81312f45f\") " pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:40.315085 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:40.315061    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:40.315085 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:40.315078    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:40.315202 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:40.315090    2577 projected.go:196] Error preparing data for projected volume kube-api-access-58fzx for pod openshift-network-diagnostics/network-check-target-jn44m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:40.315202 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:40.315147    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx podName:2c9aee2e-6b4d-4c04-9a63-d2f81312f45f nodeName:}" failed. No retries permitted until 2026-09-17 16:41:56.315130829 +0000 UTC m=+34.340039103 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-58fzx" (UniqueName: "kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx") pod "network-check-target-jn44m" (UID: "2c9aee2e-6b4d-4c04-9a63-d2f81312f45f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:40.441854 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:40.441818    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:40.441854 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:40.441858    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:40.442066 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:40.441828    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:40.442066 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:40.441941    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:40.442066 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:40.442031    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:40.442208 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:40.442114    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:42.442458 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:42.442424    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:42.442891 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:42.442544    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:42.442891 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:42.442646    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:42.442891 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:42.442778    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:42.442891 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:42.442824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:42.443110 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:42.442912    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:43.532149 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.531834    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fvkfn" event={"ID":"c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1","Type":"ContainerStarted","Data":"bd069dc7a62efd553c6b7d0ddfbc6272e2627a7950df10664a0a17666f5f29e5"}
Sep 17 16:41:43.533434 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.533410    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7dxcc" event={"ID":"a8418cea-e387-4ea4-8a0b-c69dccd07ba5","Type":"ContainerStarted","Data":"294285ad2c5b9e75766ee0f876507cc3ef940a8931993243d7775301593e828d"}
Sep 17 16:41:43.535751 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.535729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg" event={"ID":"4ec628ab-cd9d-4e35-a77c-3ba182d19487","Type":"ContainerStarted","Data":"0fd5e74ab8430ef6055024527fc9edf90282dfc0e622d03dbf873a3790bd13a4"}
Sep 17 16:41:43.535820 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.535806    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg" event={"ID":"4ec628ab-cd9d-4e35-a77c-3ba182d19487","Type":"ContainerStarted","Data":"1a96708e2a169730ef8af4fba2694abf80eb3bf23d753d9d0fd67a454686f4b4"}
Sep 17 16:41:43.535875 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.535862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg" event={"ID":"4ec628ab-cd9d-4e35-a77c-3ba182d19487","Type":"ContainerStarted","Data":"256da8d38c155d37f586ab3b7f9936764f51cc38ec1d0f53cfa640e49ac7e768"}
Sep 17 16:41:43.535926 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.535914    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg" event={"ID":"4ec628ab-cd9d-4e35-a77c-3ba182d19487","Type":"ContainerStarted","Data":"8def03370d9785ace28bc0d4e0f8698a50f0408adeb925294924e50650a27f98"}
Sep 17 16:41:43.537113 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.537095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6d9xb" event={"ID":"f4c6d895-d5c6-4540-b230-c8d40d3d4205","Type":"ContainerStarted","Data":"b6e70e6d572db37b4a434e533da6ce5742b758cd7ca635074752f6427cd06c72"}
Sep 17 16:41:43.538555 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.538536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cv7kh" event={"ID":"9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe","Type":"ContainerStarted","Data":"1e2a586da2174b79faf019afb4eee7a70ac1d358c2b97348612d47db70e39588"}
Sep 17 16:41:43.540073 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.540051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-brpkg" event={"ID":"db7116d5-b098-4a45-8f50-94a2b53e26d2","Type":"ContainerStarted","Data":"7db40645c02099fc06c63e7376e4d12eeb3656b223d18aaeca660cd884600d99"}
Sep 17 16:41:43.541539 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.541508    2577 generic.go:356] "Generic (PLEG): container finished" podID="2b2d9a82-bf68-49e0-b54a-df73639f138a" containerID="6531c43660d21d27a3882458b4229861fe719cd23d90bf01bc7b6a4470f6a239" exitCode=0
Sep 17 16:41:43.541634 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.541584    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phjts" event={"ID":"2b2d9a82-bf68-49e0-b54a-df73639f138a","Type":"ContainerDied","Data":"6531c43660d21d27a3882458b4229861fe719cd23d90bf01bc7b6a4470f6a239"}
Sep 17 16:41:43.543012 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.542934    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq" event={"ID":"bf2c6a0c-af97-4902-abe7-3a67b751c59c","Type":"ContainerStarted","Data":"ee1c140a7388dfdf565afc58f2e740e1f76e631b68c4726e04219176e922f52c"}
Sep 17 16:41:43.550468 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.550431    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fvkfn" podStartSLOduration=3.942246467 podStartE2EDuration="21.550419719s" totalImagesPullingTime="17.608173252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:43.55010126 +0000 UTC m=+21.575009547" watchObservedRunningTime="2026-09-17 16:41:43.550419719 +0000 UTC m=+21.575328004"
Sep 17 16:41:43.570838 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.570804    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cv7kh" podStartSLOduration=3.897494397 podStartE2EDuration="21.57079201s" totalImagesPullingTime="17.673297613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:43.570207123 +0000 UTC m=+21.595115411" watchObservedRunningTime="2026-09-17 16:41:43.57079201 +0000 UTC m=+21.595700311"
Sep 17 16:41:43.607102 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.607066    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-brpkg" podStartSLOduration=3.994936386 podStartE2EDuration="21.607055366s" totalImagesPullingTime="17.61211898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:43.590044292 +0000 UTC m=+21.614952582" watchObservedRunningTime="2026-09-17 16:41:43.607055366 +0000 UTC m=+21.631963653"
Sep 17 16:41:43.646852 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.645020    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7dxcc" podStartSLOduration=11.487153529 podStartE2EDuration="21.645003634s" totalImagesPullingTime="10.157850105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:43.606970403 +0000 UTC m=+21.631878691" watchObservedRunningTime="2026-09-17 16:41:43.645003634 +0000 UTC m=+21.669911922"
Sep 17 16:41:43.682573 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:43.682537    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6d9xb" podStartSLOduration=4.062248592 podStartE2EDuration="21.682524268s" totalImagesPullingTime="17.620275676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:43.682320167 +0000 UTC m=+21.707228479" watchObservedRunningTime="2026-09-17 16:41:43.682524268 +0000 UTC m=+21.707432556"
Sep 17 16:41:44.441684 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:44.441656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:44.441846 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:44.441656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:44.441846 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:44.441762    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:44.441846 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:44.441788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:44.441952 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:44.441870    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:44.441952 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:44.441931    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:44.546908 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:44.546871    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l22w8" event={"ID":"1fb52cfe-8eba-442c-8690-e4db91391c47","Type":"ContainerStarted","Data":"fb3c7749750eef1c004b4d66965d43a9daf03042185329146adc69a9007937b4"}
Sep 17 16:41:44.550634 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:44.550605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg" event={"ID":"4ec628ab-cd9d-4e35-a77c-3ba182d19487","Type":"ContainerStarted","Data":"ebcada9a2f507fc568ca162bf6a87e9dbaef97bb8d7eda142896502fbfa47b13"}
Sep 17 16:41:44.550749 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:44.550687    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg" event={"ID":"4ec628ab-cd9d-4e35-a77c-3ba182d19487","Type":"ContainerStarted","Data":"4ae363b4a2945b5ac47230210006c1d9700dcdb49b3dfd2a061fbeb9b1161d4a"}
Sep 17 16:41:44.564497 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:44.564457    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l22w8" podStartSLOduration=4.966211523 podStartE2EDuration="22.56444012s" totalImagesPullingTime="17.598228597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:44.564151399 +0000 UTC m=+22.589059688" watchObservedRunningTime="2026-09-17 16:41:44.56444012 +0000 UTC m=+22.589348409"
Sep 17 16:41:44.769225 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:44.769198    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7dxcc"
Sep 17 16:41:44.769731 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:44.769716    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7dxcc"
Sep 17 16:41:44.794584 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:44.794554    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 16:41:45.438925 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:45.438814    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T16:41:44.794571471Z","UUID":"82d5b00d-291a-480f-a1de-bb4a4c9e0eea","Handler":null,"Name":"","Endpoint":""}
Sep 17 16:41:45.440818 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:45.440794    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 16:41:45.441025 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:45.440861    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 16:41:45.554134 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:45.554103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq" event={"ID":"bf2c6a0c-af97-4902-abe7-3a67b751c59c","Type":"ContainerStarted","Data":"3592bc725c8cfcb5f733db65f62818aac1ef12b16c6d4107cb029ea611cde239"}
Sep 17 16:41:45.554643 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:45.554390    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7dxcc"
Sep 17 16:41:45.554769 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:45.554753    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7dxcc"
Sep 17 16:41:46.441393 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:46.441179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:46.441531 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:46.441180    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:46.441531 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:46.441420    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:46.441531 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:46.441216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:46.441531 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:46.441489    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:46.441685 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:46.441583    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:46.558196 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:46.558132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq" event={"ID":"bf2c6a0c-af97-4902-abe7-3a67b751c59c","Type":"ContainerStarted","Data":"8019b1bd5beefd5c6111b49500e6895d2efd5f7809c5932c4a94a94590af733a"}
Sep 17 16:41:46.561426 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:46.561403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg" event={"ID":"4ec628ab-cd9d-4e35-a77c-3ba182d19487","Type":"ContainerStarted","Data":"cdbe647a1bc41529542bf1c7f68258e23a9443810e1a7d88d5c813218d0821d3"}
Sep 17 16:41:46.575927 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:46.575887    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7xrq" podStartSLOduration=4.3689137 podStartE2EDuration="24.575875678s" totalImagesPullingTime="20.206961978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:46.575457302 +0000 UTC m=+24.600365592" watchObservedRunningTime="2026-09-17 16:41:46.575875678 +0000 UTC m=+24.600783966"
Sep 17 16:41:48.441179 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:48.441093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:48.441179 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:48.441093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:48.441602 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:48.441236    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:48.441602 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:48.441250    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:48.441602 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:48.441327    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:48.441602 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:48.441416    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:49.569355 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:49.569146    2577 generic.go:356] "Generic (PLEG): container finished" podID="2b2d9a82-bf68-49e0-b54a-df73639f138a" containerID="6b48dad9c0980b2d8fac9cd36b97e8943a3ad8db9e768c9809fd25c83e2579d6" exitCode=0
Sep 17 16:41:49.569690 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:49.569204    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phjts" event={"ID":"2b2d9a82-bf68-49e0-b54a-df73639f138a","Type":"ContainerDied","Data":"6b48dad9c0980b2d8fac9cd36b97e8943a3ad8db9e768c9809fd25c83e2579d6"}
Sep 17 16:41:49.575650 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:49.575627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg" event={"ID":"4ec628ab-cd9d-4e35-a77c-3ba182d19487","Type":"ContainerStarted","Data":"47f8de78c4bf8eceb6cd33c4ae6a3b936fab3a5eb407c944c4b0798219107ed7"}
Sep 17 16:41:49.575869 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:49.575856    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:49.575929 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:49.575913    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:49.589398 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:49.589377    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:49.617070 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:49.617029    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg" podStartSLOduration=9.980462032 podStartE2EDuration="27.617016599s" totalImagesPullingTime="17.636554567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:49.61658729 +0000 UTC m=+27.641495577" watchObservedRunningTime="2026-09-17 16:41:49.617016599 +0000 UTC m=+27.641924887"
Sep 17 16:41:50.442143 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:50.442111    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:50.442334 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:50.442248    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:50.442582 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:50.442563    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:50.442642 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:50.442629    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:50.442700 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:50.442689    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:50.442775 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:50.442759    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:50.577879 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:50.577846    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:50.591050 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:50.591024    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:41:51.050368 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:51.050341    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jn44m"]
Sep 17 16:41:51.050586 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:51.050438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:51.050586 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:51.050527    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:51.053764 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:51.053734    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9snxm"]
Sep 17 16:41:51.053875 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:51.053825    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:51.053952 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:51.053924    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:51.054399 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:51.054366    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fqmmn"]
Sep 17 16:41:51.054491 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:51.054476    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:51.054601 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:51.054580    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:51.584136 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:51.584103    2577 generic.go:356] "Generic (PLEG): container finished" podID="2b2d9a82-bf68-49e0-b54a-df73639f138a" containerID="24cd036acb030ca63711360da26f11414bf8ac65c8fcec864079295c892c67dc" exitCode=0
Sep 17 16:41:51.584742 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:51.584192    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phjts" event={"ID":"2b2d9a82-bf68-49e0-b54a-df73639f138a","Type":"ContainerDied","Data":"24cd036acb030ca63711360da26f11414bf8ac65c8fcec864079295c892c67dc"}
Sep 17 16:41:52.443047 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:52.442974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:52.443213 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:52.443125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:52.443213 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:52.443151    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:52.443351 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:52.443280    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:52.443351 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:52.443317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:52.443461 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:52.443380    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:53.589978 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:53.589819    2577 generic.go:356] "Generic (PLEG): container finished" podID="2b2d9a82-bf68-49e0-b54a-df73639f138a" containerID="d689b2a0d3542fb46833143039f74bb7db96a146e593b1371eed3b7721a1a168" exitCode=0
Sep 17 16:41:53.590368 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:53.589907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phjts" event={"ID":"2b2d9a82-bf68-49e0-b54a-df73639f138a","Type":"ContainerDied","Data":"d689b2a0d3542fb46833143039f74bb7db96a146e593b1371eed3b7721a1a168"}
Sep 17 16:41:54.441671 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:54.441639    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:54.441808 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:54.441760    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:54.441808 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:54.441778    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snxm" podUID="fd70adfb-9928-407c-b57b-d51617deb1cc"
Sep 17 16:41:54.441899 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:54.441837    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn44m" podUID="2c9aee2e-6b4d-4c04-9a63-d2f81312f45f"
Sep 17 16:41:54.441899 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:54.441860    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:54.441961 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:54.441919    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fqmmn" podUID="35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a"
Sep 17 16:41:55.799874 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.799841    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeReady"
Sep 17 16:41:55.800330 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.800002    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 16:41:55.833779 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.833723    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c7dc746d-zrqf4"]
Sep 17 16:41:55.855018 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.854998    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c7dc746d-zrqf4"]
Sep 17 16:41:55.855117 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.855041    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kz9dn"]
Sep 17 16:41:55.855151 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.855122    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:55.857554 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.857533    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 16:41:55.857748 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.857732    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 16:41:55.857937 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.857924    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 16:41:55.857974 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.857958    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nw7ln\""
Sep 17 16:41:55.869141 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.869117    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 16:41:55.872967 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.872951    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lsznc"]
Sep 17 16:41:55.873081 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.873062    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:41:55.875456 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.875436    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 16:41:55.875597 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.875580    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 16:41:55.875672 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.875660    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xc8sp\""
Sep 17 16:41:55.891210 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.891191    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lsznc"]
Sep 17 16:41:55.891295 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.891218    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kz9dn"]
Sep 17 16:41:55.891332 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.891322    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lsznc"
Sep 17 16:41:55.893742 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.893723    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 16:41:55.893843 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.893741    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jlx7p\""
Sep 17 16:41:55.893843 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.893785    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 16:41:55.893843 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.893816    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 16:41:55.935511 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.935489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58t8v\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-kube-api-access-58t8v\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:55.935600 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.935536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-bound-sa-token\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:55.935600 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.935556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4512d740-dc14-4782-a5f6-67949e27ae41-image-registry-private-configuration\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:55.935717 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.935643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:55.935717 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.935678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4512d740-dc14-4782-a5f6-67949e27ae41-trusted-ca\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:55.935819 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.935716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4512d740-dc14-4782-a5f6-67949e27ae41-ca-trust-extracted\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:55.935819 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.935776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4512d740-dc14-4782-a5f6-67949e27ae41-registry-certificates\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:55.935819 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:55.935800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4512d740-dc14-4782-a5f6-67949e27ae41-installation-pull-secrets\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.036860 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.036823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d8d1f173-79e9-40f2-adf4-221f6e68df5c-config-volume\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:41:56.036860 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.036868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4512d740-dc14-4782-a5f6-67949e27ae41-ca-trust-extracted\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.037085 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.036884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s92f9\" (UniqueName: \"kubernetes.io/projected/d8d1f173-79e9-40f2-adf4-221f6e68df5c-kube-api-access-s92f9\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:41:56.037085 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.036910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh9b2\" (UniqueName: \"kubernetes.io/projected/3eea7da6-118b-4a18-9f0c-0e509101fdec-kube-api-access-sh9b2\") pod \"ingress-canary-lsznc\" (UID: \"3eea7da6-118b-4a18-9f0c-0e509101fdec\") " pod="openshift-ingress-canary/ingress-canary-lsznc"
Sep 17 16:41:56.037085 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.036996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4512d740-dc14-4782-a5f6-67949e27ae41-registry-certificates\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.037085 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.037022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4512d740-dc14-4782-a5f6-67949e27ae41-installation-pull-secrets\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.037085 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.037058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58t8v\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-kube-api-access-58t8v\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.037384 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.037098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:41:56.037384 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.037122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3eea7da6-118b-4a18-9f0c-0e509101fdec-cert\") pod \"ingress-canary-lsznc\" (UID: \"3eea7da6-118b-4a18-9f0c-0e509101fdec\") " pod="openshift-ingress-canary/ingress-canary-lsznc"
Sep 17 16:41:56.037384 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.037155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-bound-sa-token\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.037384 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.037189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4512d740-dc14-4782-a5f6-67949e27ae41-image-registry-private-configuration\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.037384 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.037211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.037384 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.037227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4512d740-dc14-4782-a5f6-67949e27ae41-trusted-ca\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.037384 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.037243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d1f173-79e9-40f2-adf4-221f6e68df5c-tmp-dir\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:41:56.037718 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.037392    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4512d740-dc14-4782-a5f6-67949e27ae41-ca-trust-extracted\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.037718 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.037601    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4512d740-dc14-4782-a5f6-67949e27ae41-registry-certificates\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.037718 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.037696    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:41:56.037718 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.037712    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c7dc746d-zrqf4: secret "image-registry-tls" not found
Sep 17 16:41:56.037839 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.037770    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls podName:4512d740-dc14-4782-a5f6-67949e27ae41 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:56.537755235 +0000 UTC m=+34.562663501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls") pod "image-registry-7c7dc746d-zrqf4" (UID: "4512d740-dc14-4782-a5f6-67949e27ae41") : secret "image-registry-tls" not found
Sep 17 16:41:56.038601 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.038575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4512d740-dc14-4782-a5f6-67949e27ae41-trusted-ca\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.041545 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.041519    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4512d740-dc14-4782-a5f6-67949e27ae41-image-registry-private-configuration\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.041545 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.041539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4512d740-dc14-4782-a5f6-67949e27ae41-installation-pull-secrets\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.048813 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.048780    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-bound-sa-token\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.049286 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.049263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58t8v\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-kube-api-access-58t8v\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.138172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.138139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d1f173-79e9-40f2-adf4-221f6e68df5c-tmp-dir\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:41:56.138294 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.138187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d8d1f173-79e9-40f2-adf4-221f6e68df5c-config-volume\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:41:56.138294 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.138212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s92f9\" (UniqueName: \"kubernetes.io/projected/d8d1f173-79e9-40f2-adf4-221f6e68df5c-kube-api-access-s92f9\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:41:56.138294 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.138232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh9b2\" (UniqueName: \"kubernetes.io/projected/3eea7da6-118b-4a18-9f0c-0e509101fdec-kube-api-access-sh9b2\") pod \"ingress-canary-lsznc\" (UID: \"3eea7da6-118b-4a18-9f0c-0e509101fdec\") " pod="openshift-ingress-canary/ingress-canary-lsznc"
Sep 17 16:41:56.138461 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.138306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:41:56.138461 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.138330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3eea7da6-118b-4a18-9f0c-0e509101fdec-cert\") pod \"ingress-canary-lsznc\" (UID: \"3eea7da6-118b-4a18-9f0c-0e509101fdec\") " pod="openshift-ingress-canary/ingress-canary-lsznc"
Sep 17 16:41:56.138554 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.138494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d1f173-79e9-40f2-adf4-221f6e68df5c-tmp-dir\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:41:56.138554 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.138496    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:41:56.138636 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.138588    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls podName:d8d1f173-79e9-40f2-adf4-221f6e68df5c nodeName:}" failed. No retries permitted until 2026-09-17 16:41:56.638568484 +0000 UTC m=+34.663476764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls") pod "dns-default-kz9dn" (UID: "d8d1f173-79e9-40f2-adf4-221f6e68df5c") : secret "dns-default-metrics-tls" not found
Sep 17 16:41:56.138798 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.138773    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d8d1f173-79e9-40f2-adf4-221f6e68df5c-config-volume\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:41:56.140647 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.140629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3eea7da6-118b-4a18-9f0c-0e509101fdec-cert\") pod \"ingress-canary-lsznc\" (UID: \"3eea7da6-118b-4a18-9f0c-0e509101fdec\") " pod="openshift-ingress-canary/ingress-canary-lsznc"
Sep 17 16:41:56.147508 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.147485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh9b2\" (UniqueName: \"kubernetes.io/projected/3eea7da6-118b-4a18-9f0c-0e509101fdec-kube-api-access-sh9b2\") pod \"ingress-canary-lsznc\" (UID: \"3eea7da6-118b-4a18-9f0c-0e509101fdec\") " pod="openshift-ingress-canary/ingress-canary-lsznc"
Sep 17 16:41:56.147646 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.147631    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s92f9\" (UniqueName: \"kubernetes.io/projected/d8d1f173-79e9-40f2-adf4-221f6e68df5c-kube-api-access-s92f9\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:41:56.200831 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.200805    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lsznc"
Sep 17 16:41:56.239497 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.239470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:56.239604 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.239589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs\") pod \"network-metrics-daemon-fqmmn\" (UID: \"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a\") " pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:56.239674 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.239653    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:56.239731 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.239688    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:56.239731 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.239728    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret podName:fd70adfb-9928-407c-b57b-d51617deb1cc nodeName:}" failed. No retries permitted until 2026-09-17 16:42:28.239709437 +0000 UTC m=+66.264617716 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret") pod "global-pull-secret-syncer-9snxm" (UID: "fd70adfb-9928-407c-b57b-d51617deb1cc") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:56.239816 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.239751    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs podName:35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a nodeName:}" failed. No retries permitted until 2026-09-17 16:42:28.23973497 +0000 UTC m=+66.264643241 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs") pod "network-metrics-daemon-fqmmn" (UID: "35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:56.340849 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.340820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fzx\" (UniqueName: \"kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx\") pod \"network-check-target-jn44m\" (UID: \"2c9aee2e-6b4d-4c04-9a63-d2f81312f45f\") " pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:56.340986 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.340943    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:56.340986 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.340959    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:56.340986 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.340968    2577 projected.go:196] Error preparing data for projected volume kube-api-access-58fzx for pod openshift-network-diagnostics/network-check-target-jn44m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:56.341144 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.341022    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx podName:2c9aee2e-6b4d-4c04-9a63-d2f81312f45f nodeName:}" failed. No retries permitted until 2026-09-17 16:42:28.341009193 +0000 UTC m=+66.365917478 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-58fzx" (UniqueName: "kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx") pod "network-check-target-jn44m" (UID: "2c9aee2e-6b4d-4c04-9a63-d2f81312f45f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:56.442130 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.442075    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:41:56.442269 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.442074    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:41:56.442269 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.442217    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:41:56.445172 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.445144    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 16:41:56.445346 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.445273    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lng7\""
Sep 17 16:41:56.445346 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.445322    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 16:41:56.445541 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.445520    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cc7hd\""
Sep 17 16:41:56.445541 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.445538    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 16:41:56.445697 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.445550    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 16:41:56.535652 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.535625    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lsznc"]
Sep 17 16:41:56.539377 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:41:56.539345    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eea7da6_118b_4a18_9f0c_0e509101fdec.slice/crio-fa27f90f1a173fd51976aacdc74c61d7cbcf0762798c5b5a4384f729276f5015 WatchSource:0}: Error finding container fa27f90f1a173fd51976aacdc74c61d7cbcf0762798c5b5a4384f729276f5015: Status 404 returned error can't find the container with id fa27f90f1a173fd51976aacdc74c61d7cbcf0762798c5b5a4384f729276f5015
Sep 17 16:41:56.543170 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.543132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:56.543285 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.543264    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:41:56.543285 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.543286    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c7dc746d-zrqf4: secret "image-registry-tls" not found
Sep 17 16:41:56.543442 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.543336    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls podName:4512d740-dc14-4782-a5f6-67949e27ae41 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:57.543318161 +0000 UTC m=+35.568226427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls") pod "image-registry-7c7dc746d-zrqf4" (UID: "4512d740-dc14-4782-a5f6-67949e27ae41") : secret "image-registry-tls" not found
Sep 17 16:41:56.597134 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.597098    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lsznc" event={"ID":"3eea7da6-118b-4a18-9f0c-0e509101fdec","Type":"ContainerStarted","Data":"fa27f90f1a173fd51976aacdc74c61d7cbcf0762798c5b5a4384f729276f5015"}
Sep 17 16:41:56.644558 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:56.644529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:41:56.644745 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.644645    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:41:56.644745 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:56.644717    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls podName:d8d1f173-79e9-40f2-adf4-221f6e68df5c nodeName:}" failed. No retries permitted until 2026-09-17 16:41:57.644699381 +0000 UTC m=+35.669607650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls") pod "dns-default-kz9dn" (UID: "d8d1f173-79e9-40f2-adf4-221f6e68df5c") : secret "dns-default-metrics-tls" not found
Sep 17 16:41:57.553539 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:57.553357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:57.554071 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:57.553647    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:41:57.554071 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:57.553664    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c7dc746d-zrqf4: secret "image-registry-tls" not found
Sep 17 16:41:57.554071 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:57.553726    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls podName:4512d740-dc14-4782-a5f6-67949e27ae41 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:59.55370656 +0000 UTC m=+37.578614847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls") pod "image-registry-7c7dc746d-zrqf4" (UID: "4512d740-dc14-4782-a5f6-67949e27ae41") : secret "image-registry-tls" not found
Sep 17 16:41:57.654174 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:57.654128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:41:57.654339 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:57.654270    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:41:57.654339 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:57.654332    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls podName:d8d1f173-79e9-40f2-adf4-221f6e68df5c nodeName:}" failed. No retries permitted until 2026-09-17 16:41:59.654314555 +0000 UTC m=+37.679222837 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls") pod "dns-default-kz9dn" (UID: "d8d1f173-79e9-40f2-adf4-221f6e68df5c") : secret "dns-default-metrics-tls" not found
Sep 17 16:41:59.568146 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:59.568107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:41:59.568525 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:59.568253    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:41:59.568525 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:59.568273    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c7dc746d-zrqf4: secret "image-registry-tls" not found
Sep 17 16:41:59.568525 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:59.568325    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls podName:4512d740-dc14-4782-a5f6-67949e27ae41 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:03.568310051 +0000 UTC m=+41.593218322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls") pod "image-registry-7c7dc746d-zrqf4" (UID: "4512d740-dc14-4782-a5f6-67949e27ae41") : secret "image-registry-tls" not found
Sep 17 16:41:59.668704 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:41:59.668676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:41:59.668839 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:59.668822    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:41:59.668887 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:41:59.668879    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls podName:d8d1f173-79e9-40f2-adf4-221f6e68df5c nodeName:}" failed. No retries permitted until 2026-09-17 16:42:03.668864562 +0000 UTC m=+41.693772829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls") pod "dns-default-kz9dn" (UID: "d8d1f173-79e9-40f2-adf4-221f6e68df5c") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:00.607486 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:00.607451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lsznc" event={"ID":"3eea7da6-118b-4a18-9f0c-0e509101fdec","Type":"ContainerStarted","Data":"5c409bc784e1ce1292a251604be983acf89bd9cb7308ccfb534a6e0795383fcc"}
Sep 17 16:42:00.610008 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:00.609983    2577 generic.go:356] "Generic (PLEG): container finished" podID="2b2d9a82-bf68-49e0-b54a-df73639f138a" containerID="1dc746daea236381a3a10848867672bb33d9bb48d37ea21f3a2b717b0938ed8c" exitCode=0
Sep 17 16:42:00.610111 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:00.610020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phjts" event={"ID":"2b2d9a82-bf68-49e0-b54a-df73639f138a","Type":"ContainerDied","Data":"1dc746daea236381a3a10848867672bb33d9bb48d37ea21f3a2b717b0938ed8c"}
Sep 17 16:42:00.623140 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:00.623094    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lsznc" podStartSLOduration=2.4518506970000002 podStartE2EDuration="5.623076824s" totalImagesPullingTime="3.171226127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:00.622984252 +0000 UTC m=+38.647892554" watchObservedRunningTime="2026-09-17 16:42:00.623076824 +0000 UTC m=+38.647985113"
Sep 17 16:42:01.615650 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:01.615611    2577 generic.go:356] "Generic (PLEG): container finished" podID="2b2d9a82-bf68-49e0-b54a-df73639f138a" containerID="1cdd96fd157547ce9ba17c15f5c334d569f599e85df9e81e6d5720b667abc8fa" exitCode=0
Sep 17 16:42:01.616017 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:01.615692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phjts" event={"ID":"2b2d9a82-bf68-49e0-b54a-df73639f138a","Type":"ContainerDied","Data":"1cdd96fd157547ce9ba17c15f5c334d569f599e85df9e81e6d5720b667abc8fa"}
Sep 17 16:42:02.620961 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:02.620932    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phjts" event={"ID":"2b2d9a82-bf68-49e0-b54a-df73639f138a","Type":"ContainerStarted","Data":"393185ea81b75f291a2ecee54d50fce985cf115442bb77ef13f4fd9814ae409a"}
Sep 17 16:42:02.651224 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:02.651182    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-phjts" podStartSLOduration=8.931839912000001 podStartE2EDuration="40.651151023s" totalImagesPullingTime="31.719311111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:02.650607163 +0000 UTC m=+40.675515453" watchObservedRunningTime="2026-09-17 16:42:02.651151023 +0000 UTC m=+40.676059330"
Sep 17 16:42:03.596683 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:03.596637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:42:03.596838 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:03.596785    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:03.596838 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:03.596807    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c7dc746d-zrqf4: secret "image-registry-tls" not found
Sep 17 16:42:03.596903 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:03.596875    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls podName:4512d740-dc14-4782-a5f6-67949e27ae41 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:11.596859293 +0000 UTC m=+49.621767559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls") pod "image-registry-7c7dc746d-zrqf4" (UID: "4512d740-dc14-4782-a5f6-67949e27ae41") : secret "image-registry-tls" not found
Sep 17 16:42:03.697697 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:03.697665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:42:03.698058 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:03.697811    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:03.698058 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:03.697902    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls podName:d8d1f173-79e9-40f2-adf4-221f6e68df5c nodeName:}" failed. No retries permitted until 2026-09-17 16:42:11.697885146 +0000 UTC m=+49.722793413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls") pod "dns-default-kz9dn" (UID: "d8d1f173-79e9-40f2-adf4-221f6e68df5c") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:11.650620 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:11.650584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:42:11.650998 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:11.650688    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:11.650998 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:11.650699    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c7dc746d-zrqf4: secret "image-registry-tls" not found
Sep 17 16:42:11.650998 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:11.650758    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls podName:4512d740-dc14-4782-a5f6-67949e27ae41 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:27.650742177 +0000 UTC m=+65.675650443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls") pod "image-registry-7c7dc746d-zrqf4" (UID: "4512d740-dc14-4782-a5f6-67949e27ae41") : secret "image-registry-tls" not found
Sep 17 16:42:11.751127 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:11.751087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:42:11.751222 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:11.751209    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:11.751274 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:11.751265    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls podName:d8d1f173-79e9-40f2-adf4-221f6e68df5c nodeName:}" failed. No retries permitted until 2026-09-17 16:42:27.751252329 +0000 UTC m=+65.776160595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls") pod "dns-default-kz9dn" (UID: "d8d1f173-79e9-40f2-adf4-221f6e68df5c") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:17.154020 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.153985    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cfz55"]
Sep 17 16:42:17.158295 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.158275    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cfz55"
Sep 17 16:42:17.160844 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.160824    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:42:17.160923 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.160889    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-96ct2\""
Sep 17 16:42:17.161997 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.161982    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 16:42:17.167361 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.167340    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cfz55"]
Sep 17 16:42:17.253012 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.252987    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-678qx"]
Sep 17 16:42:17.256880 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.256866    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.259788 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.259770    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 16:42:17.260017 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.259998    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 16:42:17.260289 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.260273    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 16:42:17.260378 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.260367    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-sw4rq\""
Sep 17 16:42:17.260443 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.260391    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 16:42:17.267113 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.267083    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-678qx"]
Sep 17 16:42:17.268026 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.268006    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 16:42:17.291206 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.291148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmb79\" (UniqueName: \"kubernetes.io/projected/1e65ad15-c59c-4005-ace7-1122b4be237a-kube-api-access-vmb79\") pod \"volume-data-source-validator-5c4db8bdf9-cfz55\" (UID: \"1e65ad15-c59c-4005-ace7-1122b4be237a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cfz55"
Sep 17 16:42:17.343804 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.343784    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"]
Sep 17 16:42:17.347531 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.347516    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"
Sep 17 16:42:17.350097 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.350079    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 16:42:17.350173 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.350105    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:42:17.350173 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.350105    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-n4qts\""
Sep 17 16:42:17.350441 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.350426    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 16:42:17.354591 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.354573    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-sk4pl"]
Sep 17 16:42:17.358075 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.358055    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-s9jpc"]
Sep 17 16:42:17.358195 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.358179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk4pl"
Sep 17 16:42:17.360723 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.360703    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rgkq8\""
Sep 17 16:42:17.361746 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.361728    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"]
Sep 17 16:42:17.361835 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.361797    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:17.364936 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.364915    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zf6sn\""
Sep 17 16:42:17.365031 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.364974    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 16:42:17.365031 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.364990    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 16:42:17.365343 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.365324    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:42:17.365440 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.365380    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 16:42:17.369877 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.369857    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-sk4pl"]
Sep 17 16:42:17.372073 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.372054    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 16:42:17.372279 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.372095    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-s9jpc"]
Sep 17 16:42:17.392409 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.392388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmb79\" (UniqueName: \"kubernetes.io/projected/1e65ad15-c59c-4005-ace7-1122b4be237a-kube-api-access-vmb79\") pod \"volume-data-source-validator-5c4db8bdf9-cfz55\" (UID: \"1e65ad15-c59c-4005-ace7-1122b4be237a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cfz55"
Sep 17 16:42:17.392490 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.392413    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.392490 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.392429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.392490 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.392445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbg2l\" (UniqueName: \"kubernetes.io/projected/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-kube-api-access-mbg2l\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.392594 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.392498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-tmp\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.392594 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.392533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-snapshots\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.392655 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.392603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-serving-cert\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.403531 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.403506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmb79\" (UniqueName: \"kubernetes.io/projected/1e65ad15-c59c-4005-ace7-1122b4be237a-kube-api-access-vmb79\") pod \"volume-data-source-validator-5c4db8bdf9-cfz55\" (UID: \"1e65ad15-c59c-4005-ace7-1122b4be237a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cfz55"
Sep 17 16:42:17.466886 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.466840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cfz55"
Sep 17 16:42:17.493671 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.493641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-serving-cert\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.493774 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.493756    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0b353d50-4511-4e9d-a50d-362d67b3e222-serving-cert\") pod \"console-operator-fd4bb46c-s9jpc\" (UID: \"0b353d50-4511-4e9d-a50d-362d67b3e222\") " pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:17.493828 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.493787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.493828 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.493811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.493921 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.493837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbg2l\" (UniqueName: \"kubernetes.io/projected/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-kube-api-access-mbg2l\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.493921 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.493876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-tmp\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.493921 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.493899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-snapshots\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.494060 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.493923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4dr\" (UniqueName: \"kubernetes.io/projected/af8b4478-4c0f-4076-8327-261955205e93-kube-api-access-mm4dr\") pod \"network-check-source-7cbc45fddc-sk4pl\" (UID: \"af8b4478-4c0f-4076-8327-261955205e93\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk4pl"
Sep 17 16:42:17.494060 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.493948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qr9w\" (UniqueName: \"kubernetes.io/projected/72a132de-8555-4c2f-bf5b-ba3757cce3cd-kube-api-access-9qr9w\") pod \"cluster-samples-operator-685d7d68b5-tb8ch\" (UID: \"72a132de-8555-4c2f-bf5b-ba3757cce3cd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"
Sep 17 16:42:17.494060 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.493989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b353d50-4511-4e9d-a50d-362d67b3e222-config\") pod \"console-operator-fd4bb46c-s9jpc\" (UID: \"0b353d50-4511-4e9d-a50d-362d67b3e222\") " pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:17.494222 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.494071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b353d50-4511-4e9d-a50d-362d67b3e222-trusted-ca\") pod \"console-operator-fd4bb46c-s9jpc\" (UID: \"0b353d50-4511-4e9d-a50d-362d67b3e222\") " pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:17.494281 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.494224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbmw\" (UniqueName: \"kubernetes.io/projected/0b353d50-4511-4e9d-a50d-362d67b3e222-kube-api-access-6nbmw\") pod \"console-operator-fd4bb46c-s9jpc\" (UID: \"0b353d50-4511-4e9d-a50d-362d67b3e222\") " pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:17.494332 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.494287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-tb8ch\" (UID: \"72a132de-8555-4c2f-bf5b-ba3757cce3cd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"
Sep 17 16:42:17.494760 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.494729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.494887 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.494819    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-tmp\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.494887 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.494825    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-snapshots\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.495029 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.495010    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.495939 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.495920    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-serving-cert\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.516828 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.515992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbg2l\" (UniqueName: \"kubernetes.io/projected/e2fbafb9-cf8e-4b97-8971-c231a952ba1e-kube-api-access-mbg2l\") pod \"insights-operator-6f9c644dd9-678qx\" (UID: \"e2fbafb9-cf8e-4b97-8971-c231a952ba1e\") " pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.565575 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.565550    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-678qx"
Sep 17 16:42:17.594822 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.594797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0b353d50-4511-4e9d-a50d-362d67b3e222-serving-cert\") pod \"console-operator-fd4bb46c-s9jpc\" (UID: \"0b353d50-4511-4e9d-a50d-362d67b3e222\") " pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:17.594936 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.594852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4dr\" (UniqueName: \"kubernetes.io/projected/af8b4478-4c0f-4076-8327-261955205e93-kube-api-access-mm4dr\") pod \"network-check-source-7cbc45fddc-sk4pl\" (UID: \"af8b4478-4c0f-4076-8327-261955205e93\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk4pl"
Sep 17 16:42:17.594936 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.594879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qr9w\" (UniqueName: \"kubernetes.io/projected/72a132de-8555-4c2f-bf5b-ba3757cce3cd-kube-api-access-9qr9w\") pod \"cluster-samples-operator-685d7d68b5-tb8ch\" (UID: \"72a132de-8555-4c2f-bf5b-ba3757cce3cd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"
Sep 17 16:42:17.594936 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.594918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b353d50-4511-4e9d-a50d-362d67b3e222-config\") pod \"console-operator-fd4bb46c-s9jpc\" (UID: \"0b353d50-4511-4e9d-a50d-362d67b3e222\") " pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:17.595097 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.594940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b353d50-4511-4e9d-a50d-362d67b3e222-trusted-ca\") pod \"console-operator-fd4bb46c-s9jpc\" (UID: \"0b353d50-4511-4e9d-a50d-362d67b3e222\") " pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:17.595097 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.594967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbmw\" (UniqueName: \"kubernetes.io/projected/0b353d50-4511-4e9d-a50d-362d67b3e222-kube-api-access-6nbmw\") pod \"console-operator-fd4bb46c-s9jpc\" (UID: \"0b353d50-4511-4e9d-a50d-362d67b3e222\") " pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:17.595097 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.594994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-tb8ch\" (UID: \"72a132de-8555-4c2f-bf5b-ba3757cce3cd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"
Sep 17 16:42:17.595266 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:17.595132    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 16:42:17.595266 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:17.595218    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls podName:72a132de-8555-4c2f-bf5b-ba3757cce3cd nodeName:}" failed. No retries permitted until 2026-09-17 16:42:18.095193953 +0000 UTC m=+56.120102236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-tb8ch" (UID: "72a132de-8555-4c2f-bf5b-ba3757cce3cd") : secret "samples-operator-tls" not found
Sep 17 16:42:17.595750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.595725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b353d50-4511-4e9d-a50d-362d67b3e222-config\") pod \"console-operator-fd4bb46c-s9jpc\" (UID: \"0b353d50-4511-4e9d-a50d-362d67b3e222\") " pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:17.596127 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.596111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b353d50-4511-4e9d-a50d-362d67b3e222-trusted-ca\") pod \"console-operator-fd4bb46c-s9jpc\" (UID: \"0b353d50-4511-4e9d-a50d-362d67b3e222\") " pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:17.597018 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.596987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0b353d50-4511-4e9d-a50d-362d67b3e222-serving-cert\") pod \"console-operator-fd4bb46c-s9jpc\" (UID: \"0b353d50-4511-4e9d-a50d-362d67b3e222\") " pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:17.604532 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.604508    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbmw\" (UniqueName: \"kubernetes.io/projected/0b353d50-4511-4e9d-a50d-362d67b3e222-kube-api-access-6nbmw\") pod \"console-operator-fd4bb46c-s9jpc\" (UID: \"0b353d50-4511-4e9d-a50d-362d67b3e222\") " pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:17.605087 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.605065    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qr9w\" (UniqueName: \"kubernetes.io/projected/72a132de-8555-4c2f-bf5b-ba3757cce3cd-kube-api-access-9qr9w\") pod \"cluster-samples-operator-685d7d68b5-tb8ch\" (UID: \"72a132de-8555-4c2f-bf5b-ba3757cce3cd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"
Sep 17 16:42:17.605173 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.605066    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4dr\" (UniqueName: \"kubernetes.io/projected/af8b4478-4c0f-4076-8327-261955205e93-kube-api-access-mm4dr\") pod \"network-check-source-7cbc45fddc-sk4pl\" (UID: \"af8b4478-4c0f-4076-8327-261955205e93\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk4pl"
Sep 17 16:42:17.668084 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.668060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk4pl"
Sep 17 16:42:17.673901 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.673877    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:17.740915 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.740765    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cfz55"]
Sep 17 16:42:17.744141 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:42:17.744100    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e65ad15_c59c_4005_ace7_1122b4be237a.slice/crio-677f0ebcc0f32c2ea573cca196fe833f6db19be59bdf21e27a2c7e6afd265dee WatchSource:0}: Error finding container 677f0ebcc0f32c2ea573cca196fe833f6db19be59bdf21e27a2c7e6afd265dee: Status 404 returned error can't find the container with id 677f0ebcc0f32c2ea573cca196fe833f6db19be59bdf21e27a2c7e6afd265dee
Sep 17 16:42:17.835712 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.835690    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-678qx"]
Sep 17 16:42:17.839404 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:42:17.839381    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2fbafb9_cf8e_4b97_8971_c231a952ba1e.slice/crio-1c53991e5047d547bdc790ecbf6009b11301c9fb150180a1d0fa51a855f6adad WatchSource:0}: Error finding container 1c53991e5047d547bdc790ecbf6009b11301c9fb150180a1d0fa51a855f6adad: Status 404 returned error can't find the container with id 1c53991e5047d547bdc790ecbf6009b11301c9fb150180a1d0fa51a855f6adad
Sep 17 16:42:17.938238 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.938219    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-sk4pl"]
Sep 17 16:42:17.940461 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:42:17.940435    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf8b4478_4c0f_4076_8327_261955205e93.slice/crio-6ce3350a26adcdf1b25b937334112a0e3fb3b7f7e4ecfb0cdefcbb141eb30810 WatchSource:0}: Error finding container 6ce3350a26adcdf1b25b937334112a0e3fb3b7f7e4ecfb0cdefcbb141eb30810: Status 404 returned error can't find the container with id 6ce3350a26adcdf1b25b937334112a0e3fb3b7f7e4ecfb0cdefcbb141eb30810
Sep 17 16:42:17.941076 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:17.941055    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-s9jpc"]
Sep 17 16:42:17.947682 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:42:17.947650    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b353d50_4511_4e9d_a50d_362d67b3e222.slice/crio-5ccc85555ac5f88081434fc5b5a37e99ecfa7dbdb9bee0ad2569b5e96b375f48 WatchSource:0}: Error finding container 5ccc85555ac5f88081434fc5b5a37e99ecfa7dbdb9bee0ad2569b5e96b375f48: Status 404 returned error can't find the container with id 5ccc85555ac5f88081434fc5b5a37e99ecfa7dbdb9bee0ad2569b5e96b375f48
Sep 17 16:42:18.099414 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:18.099388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-tb8ch\" (UID: \"72a132de-8555-4c2f-bf5b-ba3757cce3cd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"
Sep 17 16:42:18.099587 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:18.099571    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 16:42:18.099658 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:18.099650    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls podName:72a132de-8555-4c2f-bf5b-ba3757cce3cd nodeName:}" failed. No retries permitted until 2026-09-17 16:42:19.099629416 +0000 UTC m=+57.124537697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-tb8ch" (UID: "72a132de-8555-4c2f-bf5b-ba3757cce3cd") : secret "samples-operator-tls" not found
Sep 17 16:42:18.656307 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:18.656264    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-678qx" event={"ID":"e2fbafb9-cf8e-4b97-8971-c231a952ba1e","Type":"ContainerStarted","Data":"1c53991e5047d547bdc790ecbf6009b11301c9fb150180a1d0fa51a855f6adad"}
Sep 17 16:42:18.657532 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:18.657498    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc" event={"ID":"0b353d50-4511-4e9d-a50d-362d67b3e222","Type":"ContainerStarted","Data":"5ccc85555ac5f88081434fc5b5a37e99ecfa7dbdb9bee0ad2569b5e96b375f48"}
Sep 17 16:42:18.658702 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:18.658675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk4pl" event={"ID":"af8b4478-4c0f-4076-8327-261955205e93","Type":"ContainerStarted","Data":"6ce3350a26adcdf1b25b937334112a0e3fb3b7f7e4ecfb0cdefcbb141eb30810"}
Sep 17 16:42:18.659844 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:18.659805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cfz55" event={"ID":"1e65ad15-c59c-4005-ace7-1122b4be237a","Type":"ContainerStarted","Data":"677f0ebcc0f32c2ea573cca196fe833f6db19be59bdf21e27a2c7e6afd265dee"}
Sep 17 16:42:19.105571 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:19.105531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-tb8ch\" (UID: \"72a132de-8555-4c2f-bf5b-ba3757cce3cd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"
Sep 17 16:42:19.105750 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:19.105690    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 16:42:19.105820 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:19.105755    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls podName:72a132de-8555-4c2f-bf5b-ba3757cce3cd nodeName:}" failed. No retries permitted until 2026-09-17 16:42:21.105735588 +0000 UTC m=+59.130643861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-tb8ch" (UID: "72a132de-8555-4c2f-bf5b-ba3757cce3cd") : secret "samples-operator-tls" not found
Sep 17 16:42:19.664882 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:19.664848    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cfz55" event={"ID":"1e65ad15-c59c-4005-ace7-1122b4be237a","Type":"ContainerStarted","Data":"89334e20fbdce71fb0e0697bc0d5b2c00a1667d7276a0ae86977408facf7cf0a"}
Sep 17 16:42:19.680511 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:19.680439    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cfz55" podStartSLOduration=1.304883523 podStartE2EDuration="2.68042436s" totalImagesPullingTime="1.375540837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:19.679396088 +0000 UTC m=+57.704304377" watchObservedRunningTime="2026-09-17 16:42:19.68042436 +0000 UTC m=+57.705332647"
Sep 17 16:42:21.117188 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.117133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-tb8ch\" (UID: \"72a132de-8555-4c2f-bf5b-ba3757cce3cd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"
Sep 17 16:42:21.117621 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:21.117280    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 16:42:21.117621 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:21.117356    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls podName:72a132de-8555-4c2f-bf5b-ba3757cce3cd nodeName:}" failed. No retries permitted until 2026-09-17 16:42:25.117335044 +0000 UTC m=+63.142243310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-tb8ch" (UID: "72a132de-8555-4c2f-bf5b-ba3757cce3cd") : secret "samples-operator-tls" not found
Sep 17 16:42:21.672923 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.672883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-678qx" event={"ID":"e2fbafb9-cf8e-4b97-8971-c231a952ba1e","Type":"ContainerStarted","Data":"510a100c9b03884db42b9140b9b66b4ef81c6255df2b11499a58186084a099d8"}
Sep 17 16:42:21.674804 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.674779    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/0.log"
Sep 17 16:42:21.675000 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.674818    2577 generic.go:356] "Generic (PLEG): container finished" podID="0b353d50-4511-4e9d-a50d-362d67b3e222" containerID="6f14230b57ca250de41195b713f6c66e4e6dede64c696ba1de42be66da6587f4" exitCode=255
Sep 17 16:42:21.675000 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.674851    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc" event={"ID":"0b353d50-4511-4e9d-a50d-362d67b3e222","Type":"ContainerDied","Data":"6f14230b57ca250de41195b713f6c66e4e6dede64c696ba1de42be66da6587f4"}
Sep 17 16:42:21.675110 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.675059    2577 scope.go:117] "RemoveContainer" containerID="6f14230b57ca250de41195b713f6c66e4e6dede64c696ba1de42be66da6587f4"
Sep 17 16:42:21.691847 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.691794    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-678qx" podStartSLOduration=1.8154814030000002 podStartE2EDuration="4.691775956s" totalImagesPullingTime="2.876294553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:21.69019676 +0000 UTC m=+59.715105071" watchObservedRunningTime="2026-09-17 16:42:21.691775956 +0000 UTC m=+59.716684244"
Sep 17 16:42:21.857610 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.857410    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4zxsg"]
Sep 17 16:42:21.881204 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.881154    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4zxsg"]
Sep 17 16:42:21.881357 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.881260    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:21.883985 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.883959    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 16:42:21.884124 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.884112    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-grs8h\""
Sep 17 16:42:21.884237 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.884218    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 16:42:21.923028 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.922928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-data-volume\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:21.923151 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.923041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tkpc\" (UniqueName: \"kubernetes.io/projected/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-api-access-6tkpc\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:21.923151 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.923066    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:21.923309 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.923154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:21.923309 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:21.923218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-crio-socket\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:22.023692 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.023659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tkpc\" (UniqueName: \"kubernetes.io/projected/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-api-access-6tkpc\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:22.023692 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.023694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:22.023882 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.023730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:22.023882 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.023754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-crio-socket\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:22.023882 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:22.023790    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:22.023882 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.023792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-data-volume\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:22.023882 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:22.023866    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm podName:d9a0ec35-0982-4a2a-841d-b24cf11b8e37 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:22.523845373 +0000 UTC m=+60.548753640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4zxsg" (UID: "d9a0ec35-0982-4a2a-841d-b24cf11b8e37") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:22.024049 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:22.023907    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:22.024049 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:22.023953    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls podName:d9a0ec35-0982-4a2a-841d-b24cf11b8e37 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:22.523940743 +0000 UTC m=+60.548849025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4zxsg" (UID: "d9a0ec35-0982-4a2a-841d-b24cf11b8e37") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:22.024049 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.023904    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-crio-socket\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:22.024049 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.024029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-data-volume\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:22.034503 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.034478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tkpc\" (UniqueName: \"kubernetes.io/projected/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-api-access-6tkpc\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:22.527900 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.527869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:22.528204 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.527965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:22.528204 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:22.527968    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:22.528204 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:22.528076    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm podName:d9a0ec35-0982-4a2a-841d-b24cf11b8e37 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:23.528053021 +0000 UTC m=+61.552961289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4zxsg" (UID: "d9a0ec35-0982-4a2a-841d-b24cf11b8e37") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:22.528204 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:22.528123    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:22.529356 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:22.528531    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls podName:d9a0ec35-0982-4a2a-841d-b24cf11b8e37 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:23.52831645 +0000 UTC m=+61.553224720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4zxsg" (UID: "d9a0ec35-0982-4a2a-841d-b24cf11b8e37") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:22.599110 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.599086    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2w4sg"
Sep 17 16:42:22.679582 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.679561    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/1.log"
Sep 17 16:42:22.679988 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.679972    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/0.log"
Sep 17 16:42:22.680049 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.680005    2577 generic.go:356] "Generic (PLEG): container finished" podID="0b353d50-4511-4e9d-a50d-362d67b3e222" containerID="7847261df13c59c99ce902a9df251f37bff7c4b3ef0aaccf96dbe76b77e12a36" exitCode=255
Sep 17 16:42:22.680103 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.680046    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc" event={"ID":"0b353d50-4511-4e9d-a50d-362d67b3e222","Type":"ContainerDied","Data":"7847261df13c59c99ce902a9df251f37bff7c4b3ef0aaccf96dbe76b77e12a36"}
Sep 17 16:42:22.680103 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.680096    2577 scope.go:117] "RemoveContainer" containerID="6f14230b57ca250de41195b713f6c66e4e6dede64c696ba1de42be66da6587f4"
Sep 17 16:42:22.680269 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.680254    2577 scope.go:117] "RemoveContainer" containerID="7847261df13c59c99ce902a9df251f37bff7c4b3ef0aaccf96dbe76b77e12a36"
Sep 17 16:42:22.680443 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:22.680422    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-s9jpc_openshift-console-operator(0b353d50-4511-4e9d-a50d-362d67b3e222)\"" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc" podUID="0b353d50-4511-4e9d-a50d-362d67b3e222"
Sep 17 16:42:22.681711 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.681687    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk4pl" event={"ID":"af8b4478-4c0f-4076-8327-261955205e93","Type":"ContainerStarted","Data":"8278bac37fd453928cfa85497c6a5b25fe591697bcabae72b0888ad37838a404"}
Sep 17 16:42:22.718593 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:22.718531    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk4pl" podStartSLOduration=1.234086596 podStartE2EDuration="5.718518462s" totalImagesPullingTime="4.484431866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:22.717763362 +0000 UTC m=+60.742671654" watchObservedRunningTime="2026-09-17 16:42:22.718518462 +0000 UTC m=+60.743426749"
Sep 17 16:42:23.537179 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:23.537139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:23.537506 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:23.537198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:23.537506 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:23.537257    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:23.537506 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:23.537306    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:23.537506 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:23.537310    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm podName:d9a0ec35-0982-4a2a-841d-b24cf11b8e37 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:25.537296175 +0000 UTC m=+63.562204441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4zxsg" (UID: "d9a0ec35-0982-4a2a-841d-b24cf11b8e37") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:23.537506 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:23.537361    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls podName:d9a0ec35-0982-4a2a-841d-b24cf11b8e37 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:25.537345501 +0000 UTC m=+63.562253774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4zxsg" (UID: "d9a0ec35-0982-4a2a-841d-b24cf11b8e37") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:23.685457 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:23.685433    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/1.log"
Sep 17 16:42:23.685728 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:23.685713    2577 scope.go:117] "RemoveContainer" containerID="7847261df13c59c99ce902a9df251f37bff7c4b3ef0aaccf96dbe76b77e12a36"
Sep 17 16:42:23.685858 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:23.685841    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-s9jpc_openshift-console-operator(0b353d50-4511-4e9d-a50d-362d67b3e222)\"" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc" podUID="0b353d50-4511-4e9d-a50d-362d67b3e222"
Sep 17 16:42:23.997327 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:23.997302    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fvkfn_c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1/dns-node-resolver/0.log"
Sep 17 16:42:24.597768 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:24.597737    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6d9xb_f4c6d895-d5c6-4540-b230-c8d40d3d4205/node-ca/0.log"
Sep 17 16:42:25.148197 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:25.148142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-tb8ch\" (UID: \"72a132de-8555-4c2f-bf5b-ba3757cce3cd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"
Sep 17 16:42:25.148376 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:25.148305    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 16:42:25.148419 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:25.148380    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls podName:72a132de-8555-4c2f-bf5b-ba3757cce3cd nodeName:}" failed. No retries permitted until 2026-09-17 16:42:33.148362871 +0000 UTC m=+71.173271140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-tb8ch" (UID: "72a132de-8555-4c2f-bf5b-ba3757cce3cd") : secret "samples-operator-tls" not found
Sep 17 16:42:25.398681 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:25.398617    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lsznc_3eea7da6-118b-4a18-9f0c-0e509101fdec/serve-healthcheck-canary/0.log"
Sep 17 16:42:25.551193 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:25.551167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:25.551282 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:25.551246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:25.551282 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:25.551248    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:25.551389 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:25.551308    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls podName:d9a0ec35-0982-4a2a-841d-b24cf11b8e37 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:29.551295674 +0000 UTC m=+67.576203941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4zxsg" (UID: "d9a0ec35-0982-4a2a-841d-b24cf11b8e37") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:25.551389 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:25.551314    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:25.551389 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:25.551377    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm podName:d9a0ec35-0982-4a2a-841d-b24cf11b8e37 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:29.551359461 +0000 UTC m=+67.576267744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4zxsg" (UID: "d9a0ec35-0982-4a2a-841d-b24cf11b8e37") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:27.665792 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:27.665754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:42:27.666291 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:27.665862    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:27.666291 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:27.665874    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c7dc746d-zrqf4: secret "image-registry-tls" not found
Sep 17 16:42:27.666291 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:27.665922    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls podName:4512d740-dc14-4782-a5f6-67949e27ae41 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:59.665910071 +0000 UTC m=+97.690818337 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls") pod "image-registry-7c7dc746d-zrqf4" (UID: "4512d740-dc14-4782-a5f6-67949e27ae41") : secret "image-registry-tls" not found
Sep 17 16:42:27.674240 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:27.674221    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:27.674283 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:27.674264    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:27.674573 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:27.674560    2577 scope.go:117] "RemoveContainer" containerID="7847261df13c59c99ce902a9df251f37bff7c4b3ef0aaccf96dbe76b77e12a36"
Sep 17 16:42:27.674705 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:27.674688    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-s9jpc_openshift-console-operator(0b353d50-4511-4e9d-a50d-362d67b3e222)\"" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc" podUID="0b353d50-4511-4e9d-a50d-362d67b3e222"
Sep 17 16:42:27.766532 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:27.766498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:42:27.766669 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:27.766631    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:27.766720 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:27.766673    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls podName:d8d1f173-79e9-40f2-adf4-221f6e68df5c nodeName:}" failed. No retries permitted until 2026-09-17 16:42:59.766659761 +0000 UTC m=+97.791568027 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls") pod "dns-default-kz9dn" (UID: "d8d1f173-79e9-40f2-adf4-221f6e68df5c") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:28.270293 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:28.270266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs\") pod \"network-metrics-daemon-fqmmn\" (UID: \"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a\") " pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:42:28.270446 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:28.270302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:42:28.273051 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:28.273032    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 16:42:28.273185 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:28.273155    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 16:42:28.280623 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:28.280605    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 16:42:28.280705 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:28.280653    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs podName:35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a nodeName:}" failed. No retries permitted until 2026-09-17 16:43:32.280640453 +0000 UTC m=+130.305548719 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs") pod "network-metrics-daemon-fqmmn" (UID: "35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a") : secret "metrics-daemon-secret" not found
Sep 17 16:42:28.282787 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:28.282766    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd70adfb-9928-407c-b57b-d51617deb1cc-original-pull-secret\") pod \"global-pull-secret-syncer-9snxm\" (UID: \"fd70adfb-9928-407c-b57b-d51617deb1cc\") " pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:42:28.370835 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:28.370811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fzx\" (UniqueName: \"kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx\") pod \"network-check-target-jn44m\" (UID: \"2c9aee2e-6b4d-4c04-9a63-d2f81312f45f\") " pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:42:28.373081 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:28.373060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58fzx\" (UniqueName: \"kubernetes.io/projected/2c9aee2e-6b4d-4c04-9a63-d2f81312f45f-kube-api-access-58fzx\") pod \"network-check-target-jn44m\" (UID: \"2c9aee2e-6b4d-4c04-9a63-d2f81312f45f\") " pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:42:28.557344 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:28.557326    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cc7hd\""
Sep 17 16:42:28.565054 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:28.565036    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:42:28.567284 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:28.567270    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snxm"
Sep 17 16:42:28.869185 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:28.869109    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jn44m"]
Sep 17 16:42:28.872077 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:42:28.872054    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9aee2e_6b4d_4c04_9a63_d2f81312f45f.slice/crio-99238e2cddd14996b08f5d20650670f4ca82c692f9cbbe471fe4740f48ce04b5 WatchSource:0}: Error finding container 99238e2cddd14996b08f5d20650670f4ca82c692f9cbbe471fe4740f48ce04b5: Status 404 returned error can't find the container with id 99238e2cddd14996b08f5d20650670f4ca82c692f9cbbe471fe4740f48ce04b5
Sep 17 16:42:28.893031 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:28.893008    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9snxm"]
Sep 17 16:42:28.896579 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:42:28.896559    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd70adfb_9928_407c_b57b_d51617deb1cc.slice/crio-af8d20193397292d9b6f7676ba51300824f5576a828fc88bf13deeab854b965d WatchSource:0}: Error finding container af8d20193397292d9b6f7676ba51300824f5576a828fc88bf13deeab854b965d: Status 404 returned error can't find the container with id af8d20193397292d9b6f7676ba51300824f5576a828fc88bf13deeab854b965d
Sep 17 16:42:29.580718 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:29.580691    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:29.580882 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:29.580742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:29.580882 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:29.580825    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:29.580882 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:29.580848    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:29.581045 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:29.580892    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls podName:d9a0ec35-0982-4a2a-841d-b24cf11b8e37 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:37.580877916 +0000 UTC m=+75.605786182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4zxsg" (UID: "d9a0ec35-0982-4a2a-841d-b24cf11b8e37") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:29.581045 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:29.580904    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm podName:d9a0ec35-0982-4a2a-841d-b24cf11b8e37 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:37.58089789 +0000 UTC m=+75.605806156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4zxsg" (UID: "d9a0ec35-0982-4a2a-841d-b24cf11b8e37") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:29.706690 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:29.706654    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9snxm" event={"ID":"fd70adfb-9928-407c-b57b-d51617deb1cc","Type":"ContainerStarted","Data":"af8d20193397292d9b6f7676ba51300824f5576a828fc88bf13deeab854b965d"}
Sep 17 16:42:29.708709 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:29.708682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jn44m" event={"ID":"2c9aee2e-6b4d-4c04-9a63-d2f81312f45f","Type":"ContainerStarted","Data":"6b06011660aea71b2d59c22957488974376d26a081cc5bb274b38db2c1cc5a8f"}
Sep 17 16:42:29.708832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:29.708773    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:42:29.708832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:29.708799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jn44m" event={"ID":"2c9aee2e-6b4d-4c04-9a63-d2f81312f45f","Type":"ContainerStarted","Data":"99238e2cddd14996b08f5d20650670f4ca82c692f9cbbe471fe4740f48ce04b5"}
Sep 17 16:42:29.726830 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:29.726782    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jn44m" podStartSLOduration=67.72676864 podStartE2EDuration="1m7.72676864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:29.726232312 +0000 UTC m=+67.751140615" watchObservedRunningTime="2026-09-17 16:42:29.72676864 +0000 UTC m=+67.751676929"
Sep 17 16:42:33.211010 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:33.210973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-tb8ch\" (UID: \"72a132de-8555-4c2f-bf5b-ba3757cce3cd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"
Sep 17 16:42:33.213765 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:33.213743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72a132de-8555-4c2f-bf5b-ba3757cce3cd-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-tb8ch\" (UID: \"72a132de-8555-4c2f-bf5b-ba3757cce3cd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"
Sep 17 16:42:33.258856 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:33.258826    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-n4qts\""
Sep 17 16:42:33.266180 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:33.266141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"
Sep 17 16:42:33.723662 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:33.723629    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9snxm" event={"ID":"fd70adfb-9928-407c-b57b-d51617deb1cc","Type":"ContainerStarted","Data":"c734ed86a67e1eab2d99df94614763d6fbae2049c7754037d8b8c1ac7b7cae2f"}
Sep 17 16:42:33.738408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:33.738344    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9snxm" podStartSLOduration=67.086126634 podStartE2EDuration="1m11.738331389s" totalImagesPullingTime="4.652204755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:33.738220452 +0000 UTC m=+71.763128739" watchObservedRunningTime="2026-09-17 16:42:33.738331389 +0000 UTC m=+71.763239677"
Sep 17 16:42:33.762438 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:33.762323    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch"]
Sep 17 16:42:34.728852 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:34.728813    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch" event={"ID":"72a132de-8555-4c2f-bf5b-ba3757cce3cd","Type":"ContainerStarted","Data":"3746e81ac1f1791c7a9d4a344ee8e38949490f357d6d54ccb82fe7e6676147fd"}
Sep 17 16:42:35.734476 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:35.734452    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch" event={"ID":"72a132de-8555-4c2f-bf5b-ba3757cce3cd","Type":"ContainerStarted","Data":"1d2d23d7fb2048038567ce31e8bbf800657bc582e7481ccd329efef0bc632007"}
Sep 17 16:42:36.740545 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:36.740504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch" event={"ID":"72a132de-8555-4c2f-bf5b-ba3757cce3cd","Type":"ContainerStarted","Data":"0139304571e1bb8e3de22231971c72cb6e70558d6d6bdc9d865ba7061c28401a"}
Sep 17 16:42:36.758105 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:36.758053    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-tb8ch" podStartSLOduration=17.929414605 podStartE2EDuration="19.75803586s" totalImagesPullingTime="1.828621255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:36.757181641 +0000 UTC m=+74.782089933" watchObservedRunningTime="2026-09-17 16:42:36.75803586 +0000 UTC m=+74.782944208"
Sep 17 16:42:37.642217 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:37.642174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:37.642399 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:37.642237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:37.642452 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:37.642405    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:37.642499 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:37.642486    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls podName:d9a0ec35-0982-4a2a-841d-b24cf11b8e37 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:53.64246534 +0000 UTC m=+91.667373629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4zxsg" (UID: "d9a0ec35-0982-4a2a-841d-b24cf11b8e37") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:37.642684 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:37.642661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:38.442060 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:38.442027    2577 scope.go:117] "RemoveContainer" containerID="7847261df13c59c99ce902a9df251f37bff7c4b3ef0aaccf96dbe76b77e12a36"
Sep 17 16:42:38.752148 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:38.752082    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 16:42:38.752499 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:38.752485    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/1.log"
Sep 17 16:42:38.752552 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:38.752515    2577 generic.go:356] "Generic (PLEG): container finished" podID="0b353d50-4511-4e9d-a50d-362d67b3e222" containerID="ee9cfede25b2ce9f0693e66479f3f8a86d1e3fa3882a221caa3f32a003322782" exitCode=255
Sep 17 16:42:38.752597 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:38.752565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc" event={"ID":"0b353d50-4511-4e9d-a50d-362d67b3e222","Type":"ContainerDied","Data":"ee9cfede25b2ce9f0693e66479f3f8a86d1e3fa3882a221caa3f32a003322782"}
Sep 17 16:42:38.752638 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:38.752626    2577 scope.go:117] "RemoveContainer" containerID="7847261df13c59c99ce902a9df251f37bff7c4b3ef0aaccf96dbe76b77e12a36"
Sep 17 16:42:38.752886 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:38.752868    2577 scope.go:117] "RemoveContainer" containerID="ee9cfede25b2ce9f0693e66479f3f8a86d1e3fa3882a221caa3f32a003322782"
Sep 17 16:42:38.753054 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:38.753036    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-s9jpc_openshift-console-operator(0b353d50-4511-4e9d-a50d-362d67b3e222)\"" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc" podUID="0b353d50-4511-4e9d-a50d-362d67b3e222"
Sep 17 16:42:39.757258 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:39.757230    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 16:42:47.674589 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:47.674458    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:47.674589 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:47.674525    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:42:47.675074 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:47.674817    2577 scope.go:117] "RemoveContainer" containerID="ee9cfede25b2ce9f0693e66479f3f8a86d1e3fa3882a221caa3f32a003322782"
Sep 17 16:42:47.675074 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:42:47.674970    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-s9jpc_openshift-console-operator(0b353d50-4511-4e9d-a50d-362d67b3e222)\"" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc" podUID="0b353d50-4511-4e9d-a50d-362d67b3e222"
Sep 17 16:42:51.702128 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.702094    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5968c464d4-7pkkj"]
Sep 17 16:42:51.706743 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.706726    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"]
Sep 17 16:42:51.706872 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.706855    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5968c464d4-7pkkj"
Sep 17 16:42:51.709436 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.709414    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-z2gvm\""
Sep 17 16:42:51.709660 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.709644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"
Sep 17 16:42:51.710487 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.710468    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 16:42:51.710591 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.710517    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 16:42:51.710591 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.710541    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 16:42:51.710591 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.710474    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 16:42:51.712220 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.712203    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 16:42:51.721002 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.720979    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"]
Sep 17 16:42:51.721152 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.721069    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5968c464d4-7pkkj"]
Sep 17 16:42:51.743427 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.743405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/40a5e5c7-a62e-4e45-ac27-01e2a37f0baa-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5968c464d4-7pkkj\" (UID: \"40a5e5c7-a62e-4e45-ac27-01e2a37f0baa\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5968c464d4-7pkkj"
Sep 17 16:42:51.743534 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.743433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkvgx\" (UniqueName: \"kubernetes.io/projected/40a5e5c7-a62e-4e45-ac27-01e2a37f0baa-kube-api-access-mkvgx\") pod \"managed-serviceaccount-addon-agent-5968c464d4-7pkkj\" (UID: \"40a5e5c7-a62e-4e45-ac27-01e2a37f0baa\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5968c464d4-7pkkj"
Sep 17 16:42:51.844469 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.844436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/40a5e5c7-a62e-4e45-ac27-01e2a37f0baa-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5968c464d4-7pkkj\" (UID: \"40a5e5c7-a62e-4e45-ac27-01e2a37f0baa\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5968c464d4-7pkkj"
Sep 17 16:42:51.844469 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.844470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkvgx\" (UniqueName: \"kubernetes.io/projected/40a5e5c7-a62e-4e45-ac27-01e2a37f0baa-kube-api-access-mkvgx\") pod \"managed-serviceaccount-addon-agent-5968c464d4-7pkkj\" (UID: \"40a5e5c7-a62e-4e45-ac27-01e2a37f0baa\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5968c464d4-7pkkj"
Sep 17 16:42:51.844690 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.844596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/544dd899-f6e8-46b3-921c-85f303c16aa7-klusterlet-config\") pod \"klusterlet-addon-workmgr-b69c59ccb-njr44\" (UID: \"544dd899-f6e8-46b3-921c-85f303c16aa7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"
Sep 17 16:42:51.844690 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.844630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d48gv\" (UniqueName: \"kubernetes.io/projected/544dd899-f6e8-46b3-921c-85f303c16aa7-kube-api-access-d48gv\") pod \"klusterlet-addon-workmgr-b69c59ccb-njr44\" (UID: \"544dd899-f6e8-46b3-921c-85f303c16aa7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"
Sep 17 16:42:51.844690 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.844686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/544dd899-f6e8-46b3-921c-85f303c16aa7-tmp\") pod \"klusterlet-addon-workmgr-b69c59ccb-njr44\" (UID: \"544dd899-f6e8-46b3-921c-85f303c16aa7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"
Sep 17 16:42:51.847305 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.847287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/40a5e5c7-a62e-4e45-ac27-01e2a37f0baa-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5968c464d4-7pkkj\" (UID: \"40a5e5c7-a62e-4e45-ac27-01e2a37f0baa\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5968c464d4-7pkkj"
Sep 17 16:42:51.855793 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.855769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkvgx\" (UniqueName: \"kubernetes.io/projected/40a5e5c7-a62e-4e45-ac27-01e2a37f0baa-kube-api-access-mkvgx\") pod \"managed-serviceaccount-addon-agent-5968c464d4-7pkkj\" (UID: \"40a5e5c7-a62e-4e45-ac27-01e2a37f0baa\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5968c464d4-7pkkj"
Sep 17 16:42:51.945498 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.945465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/544dd899-f6e8-46b3-921c-85f303c16aa7-klusterlet-config\") pod \"klusterlet-addon-workmgr-b69c59ccb-njr44\" (UID: \"544dd899-f6e8-46b3-921c-85f303c16aa7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"
Sep 17 16:42:51.945654 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.945501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d48gv\" (UniqueName: \"kubernetes.io/projected/544dd899-f6e8-46b3-921c-85f303c16aa7-kube-api-access-d48gv\") pod \"klusterlet-addon-workmgr-b69c59ccb-njr44\" (UID: \"544dd899-f6e8-46b3-921c-85f303c16aa7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"
Sep 17 16:42:51.945654 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.945628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/544dd899-f6e8-46b3-921c-85f303c16aa7-tmp\") pod \"klusterlet-addon-workmgr-b69c59ccb-njr44\" (UID: \"544dd899-f6e8-46b3-921c-85f303c16aa7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"
Sep 17 16:42:51.945982 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.945963    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/544dd899-f6e8-46b3-921c-85f303c16aa7-tmp\") pod \"klusterlet-addon-workmgr-b69c59ccb-njr44\" (UID: \"544dd899-f6e8-46b3-921c-85f303c16aa7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"
Sep 17 16:42:51.948019 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.947998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/544dd899-f6e8-46b3-921c-85f303c16aa7-klusterlet-config\") pod \"klusterlet-addon-workmgr-b69c59ccb-njr44\" (UID: \"544dd899-f6e8-46b3-921c-85f303c16aa7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"
Sep 17 16:42:51.956831 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:51.956784    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d48gv\" (UniqueName: \"kubernetes.io/projected/544dd899-f6e8-46b3-921c-85f303c16aa7-kube-api-access-d48gv\") pod \"klusterlet-addon-workmgr-b69c59ccb-njr44\" (UID: \"544dd899-f6e8-46b3-921c-85f303c16aa7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"
Sep 17 16:42:52.025325 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:52.025303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5968c464d4-7pkkj"
Sep 17 16:42:52.031245 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:52.031225    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"
Sep 17 16:42:52.312822 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:52.312726    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5968c464d4-7pkkj"]
Sep 17 16:42:52.315140 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:42:52.315117    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40a5e5c7_a62e_4e45_ac27_01e2a37f0baa.slice/crio-f136eabcf85e391187e9b9d8255c8406b53bca202cbb5b7937ba95fa0aeb1bab WatchSource:0}: Error finding container f136eabcf85e391187e9b9d8255c8406b53bca202cbb5b7937ba95fa0aeb1bab: Status 404 returned error can't find the container with id f136eabcf85e391187e9b9d8255c8406b53bca202cbb5b7937ba95fa0aeb1bab
Sep 17 16:42:52.322308 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:52.322289    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"]
Sep 17 16:42:52.323945 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:42:52.323928    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544dd899_f6e8_46b3_921c_85f303c16aa7.slice/crio-779b1b0a97248b1acea28c0ac6f01b150e4858f8ed53d4e24fa4dacb7ca80630 WatchSource:0}: Error finding container 779b1b0a97248b1acea28c0ac6f01b150e4858f8ed53d4e24fa4dacb7ca80630: Status 404 returned error can't find the container with id 779b1b0a97248b1acea28c0ac6f01b150e4858f8ed53d4e24fa4dacb7ca80630
Sep 17 16:42:52.798576 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:52.798536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44" event={"ID":"544dd899-f6e8-46b3-921c-85f303c16aa7","Type":"ContainerStarted","Data":"779b1b0a97248b1acea28c0ac6f01b150e4858f8ed53d4e24fa4dacb7ca80630"}
Sep 17 16:42:52.799621 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:52.799598    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5968c464d4-7pkkj" event={"ID":"40a5e5c7-a62e-4e45-ac27-01e2a37f0baa","Type":"ContainerStarted","Data":"f136eabcf85e391187e9b9d8255c8406b53bca202cbb5b7937ba95fa0aeb1bab"}
Sep 17 16:42:53.660817 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:53.660321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:53.665684 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:53.665539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9a0ec35-0982-4a2a-841d-b24cf11b8e37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zxsg\" (UID: \"d9a0ec35-0982-4a2a-841d-b24cf11b8e37\") " pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:53.694979 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:53.694951    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-grs8h\""
Sep 17 16:42:53.703586 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:53.703408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4zxsg"
Sep 17 16:42:54.011031 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:54.010680    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4zxsg"]
Sep 17 16:42:54.853907 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:42:54.853875    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a0ec35_0982_4a2a_841d_b24cf11b8e37.slice/crio-da87774efe1f3a0c6ad2aba0f3821938de330065e82e07527e7c342ea00d5442 WatchSource:0}: Error finding container da87774efe1f3a0c6ad2aba0f3821938de330065e82e07527e7c342ea00d5442: Status 404 returned error can't find the container with id da87774efe1f3a0c6ad2aba0f3821938de330065e82e07527e7c342ea00d5442
Sep 17 16:42:55.814589 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:55.814554    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4zxsg" event={"ID":"d9a0ec35-0982-4a2a-841d-b24cf11b8e37","Type":"ContainerStarted","Data":"da87774efe1f3a0c6ad2aba0f3821938de330065e82e07527e7c342ea00d5442"}
Sep 17 16:42:57.822178 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:57.822126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44" event={"ID":"544dd899-f6e8-46b3-921c-85f303c16aa7","Type":"ContainerStarted","Data":"316632695da318ea4377848072d2e3fc82b6a1b5809757fbac4465c32244d797"}
Sep 17 16:42:57.822647 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:57.822367    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"
Sep 17 16:42:57.824043 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:57.824011    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5968c464d4-7pkkj" event={"ID":"40a5e5c7-a62e-4e45-ac27-01e2a37f0baa","Type":"ContainerStarted","Data":"7ac47b777f775e410eab9cee6e7d4bde0bfce374f86b7da2b744835285098b8b"}
Sep 17 16:42:57.824210 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:57.824191    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44"
Sep 17 16:42:57.825933 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:57.825912    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4zxsg" event={"ID":"d9a0ec35-0982-4a2a-841d-b24cf11b8e37","Type":"ContainerStarted","Data":"0a322f3c546fd92937b90bd09d5236663e1c423391e9b1951dae7642a19a52c6"}
Sep 17 16:42:57.826013 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:57.825953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4zxsg" event={"ID":"d9a0ec35-0982-4a2a-841d-b24cf11b8e37","Type":"ContainerStarted","Data":"923f5efed647a9702d656c60ec442a463da95ccd017419fc07ab6e46cc1f69db"}
Sep 17 16:42:57.839017 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:57.838977    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b69c59ccb-njr44" podStartSLOduration=2.060883901 podStartE2EDuration="6.838965011s" totalImagesPullingTime="4.77808111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:57.837461436 +0000 UTC m=+95.862369725" watchObservedRunningTime="2026-09-17 16:42:57.838965011 +0000 UTC m=+95.863873291"
Sep 17 16:42:57.852418 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:57.852379    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5968c464d4-7pkkj" podStartSLOduration=2.129038431 podStartE2EDuration="6.852365622s" totalImagesPullingTime="4.723327191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:57.851687525 +0000 UTC m=+95.876595814" watchObservedRunningTime="2026-09-17 16:42:57.852365622 +0000 UTC m=+95.877273910"
Sep 17 16:42:59.712386 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:59.712361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:42:59.714579 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:59.714561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls\") pod \"image-registry-7c7dc746d-zrqf4\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") " pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:42:59.771919 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:59.771895    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nw7ln\""
Sep 17 16:42:59.779552 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:59.779532    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:42:59.813570 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:59.813537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:42:59.816526 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:59.816497    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8d1f173-79e9-40f2-adf4-221f6e68df5c-metrics-tls\") pod \"dns-default-kz9dn\" (UID: \"d8d1f173-79e9-40f2-adf4-221f6e68df5c\") " pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:42:59.834954 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:59.834924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4zxsg" event={"ID":"d9a0ec35-0982-4a2a-841d-b24cf11b8e37","Type":"ContainerStarted","Data":"8368ec1ad3b091d8a903f8994b62e5cc218c71bd3d69074ac5b767e0eafc5c5b"}
Sep 17 16:42:59.869819 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:42:59.869769    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4zxsg" podStartSLOduration=36.960137778 podStartE2EDuration="38.869751716s" totalImagesPullingTime="1.909613938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:59.869195986 +0000 UTC m=+97.894104274" watchObservedRunningTime="2026-09-17 16:42:59.869751716 +0000 UTC m=+97.894660004"
Sep 17 16:43:00.053738 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.053555    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c7dc746d-zrqf4"]
Sep 17 16:43:00.056056 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:43:00.056034    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4512d740_dc14_4782_a5f6_67949e27ae41.slice/crio-4a09d0c63e9e3d70e158ce50fbe7e7c666dc4d434791f4bb3ac9b8a88f015819 WatchSource:0}: Error finding container 4a09d0c63e9e3d70e158ce50fbe7e7c666dc4d434791f4bb3ac9b8a88f015819: Status 404 returned error can't find the container with id 4a09d0c63e9e3d70e158ce50fbe7e7c666dc4d434791f4bb3ac9b8a88f015819
Sep 17 16:43:00.085503 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.085478    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xc8sp\""
Sep 17 16:43:00.093849 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.093827    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:43:00.369127 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.369049    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kz9dn"]
Sep 17 16:43:00.372010 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:43:00.371980    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d1f173_79e9_40f2_adf4_221f6e68df5c.slice/crio-aaf9ba69a2f98d9b333a51edcd25ac62503ac01c5e1e171770cb1ce76aaa4d23 WatchSource:0}: Error finding container aaf9ba69a2f98d9b333a51edcd25ac62503ac01c5e1e171770cb1ce76aaa4d23: Status 404 returned error can't find the container with id aaf9ba69a2f98d9b333a51edcd25ac62503ac01c5e1e171770cb1ce76aaa4d23
Sep 17 16:43:00.714563 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.714489    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jn44m"
Sep 17 16:43:00.745971 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.745946    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gqd7m"]
Sep 17 16:43:00.750609 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.750591    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.753427 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.753366    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tw48h\""
Sep 17 16:43:00.753576 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.753561    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 16:43:00.753576 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.753570    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 16:43:00.753763 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.753602    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 16:43:00.753893 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.753841    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 16:43:00.753893 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.753855    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 16:43:00.754070 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.754055    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 16:43:00.823229 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.823202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/32b46210-87b1-43c6-aa0e-ca201d6e8097-node-exporter-textfile\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.823363 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.823258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/32b46210-87b1-43c6-aa0e-ca201d6e8097-node-exporter-wtmp\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.823363 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.823288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32b46210-87b1-43c6-aa0e-ca201d6e8097-node-exporter-tls\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.823363 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.823328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32b46210-87b1-43c6-aa0e-ca201d6e8097-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.823522 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.823437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32b46210-87b1-43c6-aa0e-ca201d6e8097-sys\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.823522 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.823473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/32b46210-87b1-43c6-aa0e-ca201d6e8097-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.823522 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.823501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbmvd\" (UniqueName: \"kubernetes.io/projected/32b46210-87b1-43c6-aa0e-ca201d6e8097-kube-api-access-mbmvd\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.823662 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.823547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/32b46210-87b1-43c6-aa0e-ca201d6e8097-root\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.823662 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.823574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32b46210-87b1-43c6-aa0e-ca201d6e8097-metrics-client-ca\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.839651 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.839618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kz9dn" event={"ID":"d8d1f173-79e9-40f2-adf4-221f6e68df5c","Type":"ContainerStarted","Data":"aaf9ba69a2f98d9b333a51edcd25ac62503ac01c5e1e171770cb1ce76aaa4d23"}
Sep 17 16:43:00.841298 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.841275    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4" event={"ID":"4512d740-dc14-4782-a5f6-67949e27ae41","Type":"ContainerStarted","Data":"6179a0dfc8132d41bcde27a03456ffb6ad8d377b05ce9b1ce7d8dc0383959bc9"}
Sep 17 16:43:00.841407 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.841313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4" event={"ID":"4512d740-dc14-4782-a5f6-67949e27ae41","Type":"ContainerStarted","Data":"4a09d0c63e9e3d70e158ce50fbe7e7c666dc4d434791f4bb3ac9b8a88f015819"}
Sep 17 16:43:00.867513 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.867465    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4" podStartSLOduration=97.867448921 podStartE2EDuration="1m37.867448921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:00.86651326 +0000 UTC m=+98.891421577" watchObservedRunningTime="2026-09-17 16:43:00.867448921 +0000 UTC m=+98.892357201"
Sep 17 16:43:00.924171 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.924140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/32b46210-87b1-43c6-aa0e-ca201d6e8097-root\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.924273 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.924189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32b46210-87b1-43c6-aa0e-ca201d6e8097-metrics-client-ca\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.924273 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.924224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/32b46210-87b1-43c6-aa0e-ca201d6e8097-node-exporter-textfile\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.924273 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.924227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/32b46210-87b1-43c6-aa0e-ca201d6e8097-root\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.924484 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.924455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/32b46210-87b1-43c6-aa0e-ca201d6e8097-node-exporter-wtmp\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.924566 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.924533    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/32b46210-87b1-43c6-aa0e-ca201d6e8097-node-exporter-textfile\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.924910 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.924889    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/32b46210-87b1-43c6-aa0e-ca201d6e8097-node-exporter-wtmp\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.924972 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.924954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32b46210-87b1-43c6-aa0e-ca201d6e8097-node-exporter-tls\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.925017 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.924994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32b46210-87b1-43c6-aa0e-ca201d6e8097-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.925064 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.925041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32b46210-87b1-43c6-aa0e-ca201d6e8097-sys\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.925102 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.925061    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32b46210-87b1-43c6-aa0e-ca201d6e8097-metrics-client-ca\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.925102 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.925068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/32b46210-87b1-43c6-aa0e-ca201d6e8097-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.925102 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.925098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbmvd\" (UniqueName: \"kubernetes.io/projected/32b46210-87b1-43c6-aa0e-ca201d6e8097-kube-api-access-mbmvd\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.925676 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.925484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32b46210-87b1-43c6-aa0e-ca201d6e8097-sys\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.925676 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.925594    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/32b46210-87b1-43c6-aa0e-ca201d6e8097-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.927813 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.927793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32b46210-87b1-43c6-aa0e-ca201d6e8097-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.927936 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.927912    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32b46210-87b1-43c6-aa0e-ca201d6e8097-node-exporter-tls\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:00.934518 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:00.934496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbmvd\" (UniqueName: \"kubernetes.io/projected/32b46210-87b1-43c6-aa0e-ca201d6e8097-kube-api-access-mbmvd\") pod \"node-exporter-gqd7m\" (UID: \"32b46210-87b1-43c6-aa0e-ca201d6e8097\") " pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:01.060815 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:01.060783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gqd7m"
Sep 17 16:43:01.073847 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:43:01.073815    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b46210_87b1_43c6_aa0e_ca201d6e8097.slice/crio-40021b3e5258d8f7fbd661a67e7b0ba700392d8fc0af790928a1c06b2b0b3101 WatchSource:0}: Error finding container 40021b3e5258d8f7fbd661a67e7b0ba700392d8fc0af790928a1c06b2b0b3101: Status 404 returned error can't find the container with id 40021b3e5258d8f7fbd661a67e7b0ba700392d8fc0af790928a1c06b2b0b3101
Sep 17 16:43:01.846391 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:01.846342    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqd7m" event={"ID":"32b46210-87b1-43c6-aa0e-ca201d6e8097","Type":"ContainerStarted","Data":"40021b3e5258d8f7fbd661a67e7b0ba700392d8fc0af790928a1c06b2b0b3101"}
Sep 17 16:43:01.846755 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:01.846545    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:43:02.443017 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.442989    2577 scope.go:117] "RemoveContainer" containerID="ee9cfede25b2ce9f0693e66479f3f8a86d1e3fa3882a221caa3f32a003322782"
Sep 17 16:43:02.738366 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.738300    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57648dc9b4-mbl62"]
Sep 17 16:43:02.741973 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.741951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.745877 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.745847    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 16:43:02.746219 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.746199    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 16:43:02.746480 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.746459    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gtmzx\""
Sep 17 16:43:02.746766 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.746743    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4npu0bmb63v5o\""
Sep 17 16:43:02.747037 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.747020    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 16:43:02.747310 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.747293    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 16:43:02.747400 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.747317    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 16:43:02.756647 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.756619    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57648dc9b4-mbl62"]
Sep 17 16:43:02.842824 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.842797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-thanos-querier-tls\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.842980 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.842828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.842980 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.842850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-grpc-tls\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.842980 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.842899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.842980 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.842944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.843216 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.843058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k9cb\" (UniqueName: \"kubernetes.io/projected/29d15cf0-b377-48e4-99a9-ff34677d7b82-kube-api-access-6k9cb\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.843216 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.843102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.843216 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.843192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29d15cf0-b377-48e4-99a9-ff34677d7b82-metrics-client-ca\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.852234 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.852202    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kz9dn" event={"ID":"d8d1f173-79e9-40f2-adf4-221f6e68df5c","Type":"ContainerStarted","Data":"03f0ba8ce294936280ae8105d3e0f3c7ecd0a6df077462210402a095036d2e19"}
Sep 17 16:43:02.852624 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.852267    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kz9dn" event={"ID":"d8d1f173-79e9-40f2-adf4-221f6e68df5c","Type":"ContainerStarted","Data":"7fae2c33f58fca07793d3c6eff208d1492356b3878e8689f86fe6d4dbfc5df61"}
Sep 17 16:43:02.852624 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.852300    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:43:02.856595 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.854610    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 16:43:02.856595 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.854707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc" event={"ID":"0b353d50-4511-4e9d-a50d-362d67b3e222","Type":"ContainerStarted","Data":"0c015a85eb624467c51b7b19b7b2657f38552497bfb24a9c510b63ff62b3af47"}
Sep 17 16:43:02.856595 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.856421    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:43:02.858841 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.858814    2577 generic.go:356] "Generic (PLEG): container finished" podID="32b46210-87b1-43c6-aa0e-ca201d6e8097" containerID="8bf4c808002e69318e97dcdb91abe23f0d319b8b0f89147f69b0889b07ddd549" exitCode=0
Sep 17 16:43:02.858957 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.858911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqd7m" event={"ID":"32b46210-87b1-43c6-aa0e-ca201d6e8097","Type":"ContainerDied","Data":"8bf4c808002e69318e97dcdb91abe23f0d319b8b0f89147f69b0889b07ddd549"}
Sep 17 16:43:02.863219 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.863200    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc"
Sep 17 16:43:02.910690 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.910644    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kz9dn" podStartSLOduration=66.384037038 podStartE2EDuration="1m7.910625676s" totalImagesPullingTime="1.526588638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:02.88198851 +0000 UTC m=+100.906896802" watchObservedRunningTime="2026-09-17 16:43:02.910625676 +0000 UTC m=+100.935533964"
Sep 17 16:43:02.911434 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.911397    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-s9jpc" podStartSLOduration=43.141097582 podStartE2EDuration="45.911381581s" totalImagesPullingTime="2.770283999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:02.9098833 +0000 UTC m=+100.934791588" watchObservedRunningTime="2026-09-17 16:43:02.911381581 +0000 UTC m=+100.936289870"
Sep 17 16:43:02.914017 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.913999    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-dpwv7"]
Sep 17 16:43:02.920616 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.920599    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-dpwv7"
Sep 17 16:43:02.923736 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.923718    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 16:43:02.924343 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.924325    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 16:43:02.924442 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.924329    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-jfn4p\""
Sep 17 16:43:02.938962 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.938939    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-dpwv7"]
Sep 17 16:43:02.944339 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.944315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.944841 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.944813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29d15cf0-b377-48e4-99a9-ff34677d7b82-metrics-client-ca\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.944990 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.944942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-thanos-querier-tls\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.944990 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.944972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.945112 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.944997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-grpc-tls\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.945112 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.945090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.945235 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.945140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.945773 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.945295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k9cb\" (UniqueName: \"kubernetes.io/projected/29d15cf0-b377-48e4-99a9-ff34677d7b82-kube-api-access-6k9cb\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.945855 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.945817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29d15cf0-b377-48e4-99a9-ff34677d7b82-metrics-client-ca\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.949896 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.949857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.953177 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.951455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-grpc-tls\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.953177 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.952131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.953177 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.952874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-thanos-querier-tls\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.953177 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.952989    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.953663 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.953644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29d15cf0-b377-48e4-99a9-ff34677d7b82-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:02.969552 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:02.969527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k9cb\" (UniqueName: \"kubernetes.io/projected/29d15cf0-b377-48e4-99a9-ff34677d7b82-kube-api-access-6k9cb\") pod \"thanos-querier-57648dc9b4-mbl62\" (UID: \"29d15cf0-b377-48e4-99a9-ff34677d7b82\") " pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:03.045943 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:03.045917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwlrh\" (UniqueName: \"kubernetes.io/projected/de57cca8-b211-4c6d-8691-93914d97d54f-kube-api-access-cwlrh\") pod \"downloads-6594f4c75-dpwv7\" (UID: \"de57cca8-b211-4c6d-8691-93914d97d54f\") " pod="openshift-console/downloads-6594f4c75-dpwv7"
Sep 17 16:43:03.055841 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:03.055820    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:03.146677 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:03.146650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwlrh\" (UniqueName: \"kubernetes.io/projected/de57cca8-b211-4c6d-8691-93914d97d54f-kube-api-access-cwlrh\") pod \"downloads-6594f4c75-dpwv7\" (UID: \"de57cca8-b211-4c6d-8691-93914d97d54f\") " pod="openshift-console/downloads-6594f4c75-dpwv7"
Sep 17 16:43:03.156715 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:03.156695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwlrh\" (UniqueName: \"kubernetes.io/projected/de57cca8-b211-4c6d-8691-93914d97d54f-kube-api-access-cwlrh\") pod \"downloads-6594f4c75-dpwv7\" (UID: \"de57cca8-b211-4c6d-8691-93914d97d54f\") " pod="openshift-console/downloads-6594f4c75-dpwv7"
Sep 17 16:43:03.229635 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:03.229597    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-dpwv7"
Sep 17 16:43:03.334023 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:03.333994    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57648dc9b4-mbl62"]
Sep 17 16:43:03.337401 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:43:03.337370    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d15cf0_b377_48e4_99a9_ff34677d7b82.slice/crio-6fe990b1b56e6effa988bdf4f567d1de608fec2a5a9776ff366d4af5d1bb9466 WatchSource:0}: Error finding container 6fe990b1b56e6effa988bdf4f567d1de608fec2a5a9776ff366d4af5d1bb9466: Status 404 returned error can't find the container with id 6fe990b1b56e6effa988bdf4f567d1de608fec2a5a9776ff366d4af5d1bb9466
Sep 17 16:43:03.502317 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:03.502296    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-dpwv7"]
Sep 17 16:43:03.504615 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:43:03.504587    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde57cca8_b211_4c6d_8691_93914d97d54f.slice/crio-17be17cdfc9a8e5ce7178a3b3102af18ec57429580ffbd2c4942aa5e49e3a3b9 WatchSource:0}: Error finding container 17be17cdfc9a8e5ce7178a3b3102af18ec57429580ffbd2c4942aa5e49e3a3b9: Status 404 returned error can't find the container with id 17be17cdfc9a8e5ce7178a3b3102af18ec57429580ffbd2c4942aa5e49e3a3b9
Sep 17 16:43:03.868230 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:03.868188    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-dpwv7" event={"ID":"de57cca8-b211-4c6d-8691-93914d97d54f","Type":"ContainerStarted","Data":"17be17cdfc9a8e5ce7178a3b3102af18ec57429580ffbd2c4942aa5e49e3a3b9"}
Sep 17 16:43:03.869504 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:03.869479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62" event={"ID":"29d15cf0-b377-48e4-99a9-ff34677d7b82","Type":"ContainerStarted","Data":"6fe990b1b56e6effa988bdf4f567d1de608fec2a5a9776ff366d4af5d1bb9466"}
Sep 17 16:43:03.871640 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:03.871614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqd7m" event={"ID":"32b46210-87b1-43c6-aa0e-ca201d6e8097","Type":"ContainerStarted","Data":"9367104d0b94fcf5648d4007e3bec4c200a44ce87c5343dfe59f72aabda4bbff"}
Sep 17 16:43:03.871729 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:03.871660    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqd7m" event={"ID":"32b46210-87b1-43c6-aa0e-ca201d6e8097","Type":"ContainerStarted","Data":"dd574ac70d3eacb826ca3c3f9e86cf4d3499e91b5e213f462c1c501f0eceae8c"}
Sep 17 16:43:03.891747 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:03.891711    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gqd7m" podStartSLOduration=2.485760489 podStartE2EDuration="3.891700184s" totalImagesPullingTime="1.340268373s" totalInitContainerRuntime="65.671322ms" isStatefulPod=true podCreationTimestamp="2026-09-17 16:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:03.890979995 +0000 UTC m=+101.915888284" watchObservedRunningTime="2026-09-17 16:43:03.891700184 +0000 UTC m=+101.916608472"
Sep 17 16:43:05.510239 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:05.509513    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5"]
Sep 17 16:43:05.513476 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:05.513451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5"
Sep 17 16:43:05.517449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:05.516865    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 16:43:05.517449 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:05.517085    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-l425g\""
Sep 17 16:43:05.522776 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:05.522636    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5"]
Sep 17 16:43:05.669551 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:05.669513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b03c9158-2b71-4a29-a557-823ddc1bd610-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-lvqv5\" (UID: \"b03c9158-2b71-4a29-a557-823ddc1bd610\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5"
Sep 17 16:43:05.669709 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:05.669624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rbw6\" (UniqueName: \"kubernetes.io/projected/b03c9158-2b71-4a29-a557-823ddc1bd610-kube-api-access-6rbw6\") pod \"monitoring-plugin-5984459ff8-lvqv5\" (UID: \"b03c9158-2b71-4a29-a557-823ddc1bd610\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5"
Sep 17 16:43:05.770709 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:05.770477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rbw6\" (UniqueName: \"kubernetes.io/projected/b03c9158-2b71-4a29-a557-823ddc1bd610-kube-api-access-6rbw6\") pod \"monitoring-plugin-5984459ff8-lvqv5\" (UID: \"b03c9158-2b71-4a29-a557-823ddc1bd610\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5"
Sep 17 16:43:05.770709 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:05.770628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b03c9158-2b71-4a29-a557-823ddc1bd610-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-lvqv5\" (UID: \"b03c9158-2b71-4a29-a557-823ddc1bd610\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5"
Sep 17 16:43:05.770943 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:43:05.770750    2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 16:43:05.770943 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:43:05.770810    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b03c9158-2b71-4a29-a557-823ddc1bd610-monitoring-plugin-cert podName:b03c9158-2b71-4a29-a557-823ddc1bd610 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:06.270790956 +0000 UTC m=+104.295699225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/b03c9158-2b71-4a29-a557-823ddc1bd610-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-lvqv5" (UID: "b03c9158-2b71-4a29-a557-823ddc1bd610") : secret "monitoring-plugin-cert" not found
Sep 17 16:43:05.780276 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:05.780243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rbw6\" (UniqueName: \"kubernetes.io/projected/b03c9158-2b71-4a29-a557-823ddc1bd610-kube-api-access-6rbw6\") pod \"monitoring-plugin-5984459ff8-lvqv5\" (UID: \"b03c9158-2b71-4a29-a557-823ddc1bd610\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5"
Sep 17 16:43:06.274718 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:06.274690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b03c9158-2b71-4a29-a557-823ddc1bd610-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-lvqv5\" (UID: \"b03c9158-2b71-4a29-a557-823ddc1bd610\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5"
Sep 17 16:43:06.277445 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:06.277424    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b03c9158-2b71-4a29-a557-823ddc1bd610-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-lvqv5\" (UID: \"b03c9158-2b71-4a29-a557-823ddc1bd610\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5"
Sep 17 16:43:06.427727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:06.427572    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5"
Sep 17 16:43:06.717375 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:06.717296    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5"]
Sep 17 16:43:06.721083 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:43:06.721039    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03c9158_2b71_4a29_a557_823ddc1bd610.slice/crio-0f440ec32379098d571b30f9ecf9ed1c1063f6b105f63293b369c4e29aad1875 WatchSource:0}: Error finding container 0f440ec32379098d571b30f9ecf9ed1c1063f6b105f63293b369c4e29aad1875: Status 404 returned error can't find the container with id 0f440ec32379098d571b30f9ecf9ed1c1063f6b105f63293b369c4e29aad1875
Sep 17 16:43:06.886583 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:06.886549    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62" event={"ID":"29d15cf0-b377-48e4-99a9-ff34677d7b82","Type":"ContainerStarted","Data":"9257b30e111eed5a78810b35d2ee62fa5aa24ac7a36082714aee3b4d3e30254e"}
Sep 17 16:43:06.886742 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:06.886614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62" event={"ID":"29d15cf0-b377-48e4-99a9-ff34677d7b82","Type":"ContainerStarted","Data":"a9e52cffd3c6d427d976f93d4e8c2884a972c984439067453a3181152f7657a4"}
Sep 17 16:43:06.886742 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:06.886656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62" event={"ID":"29d15cf0-b377-48e4-99a9-ff34677d7b82","Type":"ContainerStarted","Data":"a89a336c7ef83c6aa50c07202f3948aa65056c9e922b488f2167db31da85cad2"}
Sep 17 16:43:06.887814 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:06.887784    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5" event={"ID":"b03c9158-2b71-4a29-a557-823ddc1bd610","Type":"ContainerStarted","Data":"0f440ec32379098d571b30f9ecf9ed1c1063f6b105f63293b369c4e29aad1875"}
Sep 17 16:43:07.897622 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:07.897545    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62" event={"ID":"29d15cf0-b377-48e4-99a9-ff34677d7b82","Type":"ContainerStarted","Data":"1fa5155060575598ea276c12d45aa7a61b24f36124a9d6ee08e87e0e88ff7571"}
Sep 17 16:43:07.898007 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:07.897673    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62" event={"ID":"29d15cf0-b377-48e4-99a9-ff34677d7b82","Type":"ContainerStarted","Data":"ceb31d4b9a3f4bf76086cc334f4eca4df08d71d1c0b4c2bb8a214fad89d3078c"}
Sep 17 16:43:08.903855 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:08.903725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5" event={"ID":"b03c9158-2b71-4a29-a557-823ddc1bd610","Type":"ContainerStarted","Data":"b99760e851042abcf88c09cdefd9a3de28f5f6d74ff2f3b9b10b141772b18cd7"}
Sep 17 16:43:08.903855 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:08.903796    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5"
Sep 17 16:43:08.908778 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:08.908241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62" event={"ID":"29d15cf0-b377-48e4-99a9-ff34677d7b82","Type":"ContainerStarted","Data":"b2864cb6189c854f53d60338d73aa66dcc27073543ede46a27558b28876aff28"}
Sep 17 16:43:08.908778 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:08.908449    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:08.910390 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:08.910373    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5"
Sep 17 16:43:08.919177 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:08.919117    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lvqv5" podStartSLOduration=1.86989595 podStartE2EDuration="3.919102178s" totalImagesPullingTime="2.049206228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:08.919056788 +0000 UTC m=+106.943965079" watchObservedRunningTime="2026-09-17 16:43:08.919102178 +0000 UTC m=+106.944010466"
Sep 17 16:43:08.941793 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:08.941746    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62" podStartSLOduration=2.883940026 podStartE2EDuration="6.941729875s" totalImagesPullingTime="4.057789849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:08.940557155 +0000 UTC m=+106.965465462" watchObservedRunningTime="2026-09-17 16:43:08.941729875 +0000 UTC m=+106.966638167"
Sep 17 16:43:12.874414 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:12.874381    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kz9dn"
Sep 17 16:43:12.921184 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:12.921134    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74596f4558-4xsl9"]
Sep 17 16:43:12.925790 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:12.925763    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:12.928909 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:12.928546    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 16:43:12.928909 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:12.928546    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 16:43:12.928909 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:12.928697    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 16:43:12.928909 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:12.928798    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 16:43:12.928909 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:12.928819    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mbrl8\""
Sep 17 16:43:12.929236 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:12.929201    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 16:43:12.936021 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:12.935965    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74596f4558-4xsl9"]
Sep 17 16:43:13.029869 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.029835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1d5c238-a754-4bc4-993b-362181e94ca3-console-serving-cert\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.029869 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.029876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-console-config\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.030107 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.029947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdqq8\" (UniqueName: \"kubernetes.io/projected/a1d5c238-a754-4bc4-993b-362181e94ca3-kube-api-access-zdqq8\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.030107 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.029977    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-service-ca\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.030107 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.030036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1d5c238-a754-4bc4-993b-362181e94ca3-console-oauth-config\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.030291 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.030175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-oauth-serving-cert\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.130684 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.130605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1d5c238-a754-4bc4-993b-362181e94ca3-console-serving-cert\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.130684 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.130652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-console-config\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.130882 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.130704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdqq8\" (UniqueName: \"kubernetes.io/projected/a1d5c238-a754-4bc4-993b-362181e94ca3-kube-api-access-zdqq8\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.130882 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.130731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-service-ca\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.130882 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.130750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1d5c238-a754-4bc4-993b-362181e94ca3-console-oauth-config\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.130882 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.130775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-oauth-serving-cert\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.131589 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.131565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-oauth-serving-cert\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.131728 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.131609    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-service-ca\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.131796 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.131726    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-console-config\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.133574 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.133553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1d5c238-a754-4bc4-993b-362181e94ca3-console-serving-cert\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.133680 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.133593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1d5c238-a754-4bc4-993b-362181e94ca3-console-oauth-config\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.139062 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.139034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdqq8\" (UniqueName: \"kubernetes.io/projected/a1d5c238-a754-4bc4-993b-362181e94ca3-kube-api-access-zdqq8\") pod \"console-74596f4558-4xsl9\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") " pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:13.238396 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:13.238362    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:14.306416 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:14.306384    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c7dc746d-zrqf4"]
Sep 17 16:43:14.919564 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:14.919534    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57648dc9b4-mbl62"
Sep 17 16:43:19.826951 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:19.826883    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74596f4558-4xsl9"]
Sep 17 16:43:19.830329 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:43:19.830301    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d5c238_a754_4bc4_993b_362181e94ca3.slice/crio-62981ea011e8289258506b1782af2e50c1c91973ec74f21041d7a393fd4849d5 WatchSource:0}: Error finding container 62981ea011e8289258506b1782af2e50c1c91973ec74f21041d7a393fd4849d5: Status 404 returned error can't find the container with id 62981ea011e8289258506b1782af2e50c1c91973ec74f21041d7a393fd4849d5
Sep 17 16:43:19.959306 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:19.959255    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74596f4558-4xsl9" event={"ID":"a1d5c238-a754-4bc4-993b-362181e94ca3","Type":"ContainerStarted","Data":"62981ea011e8289258506b1782af2e50c1c91973ec74f21041d7a393fd4849d5"}
Sep 17 16:43:19.961349 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:19.961318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-dpwv7" event={"ID":"de57cca8-b211-4c6d-8691-93914d97d54f","Type":"ContainerStarted","Data":"3f097827cda18dfeb0b8b14bac2a981fac108fc92caf788df1e7aa6e28dfb4e4"}
Sep 17 16:43:19.961603 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:19.961579    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-dpwv7"
Sep 17 16:43:19.977990 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:19.977963    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-dpwv7"
Sep 17 16:43:19.978143 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:19.978094    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-dpwv7" podStartSLOduration=1.831660045 podStartE2EDuration="17.978075848s" totalImagesPullingTime="16.146415803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:19.976853049 +0000 UTC m=+118.001761338" watchObservedRunningTime="2026-09-17 16:43:19.978075848 +0000 UTC m=+118.002984138"
Sep 17 16:43:24.313969 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:24.313941    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:43:26.002053 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:26.002014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74596f4558-4xsl9" event={"ID":"a1d5c238-a754-4bc4-993b-362181e94ca3","Type":"ContainerStarted","Data":"1c0868ef87827ff0f29d322e2a13a30a9a20f8ae51a5793056952dd9466b1db8"}
Sep 17 16:43:26.033587 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:26.033528    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74596f4558-4xsl9" podStartSLOduration=8.005506346 podStartE2EDuration="14.033505854s" totalImagesPullingTime="6.027999508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:26.03115123 +0000 UTC m=+124.056059512" watchObservedRunningTime="2026-09-17 16:43:26.033505854 +0000 UTC m=+124.058414143"
Sep 17 16:43:31.494230 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:31.494201    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74596f4558-4xsl9"]
Sep 17 16:43:32.296485 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:32.296446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs\") pod \"network-metrics-daemon-fqmmn\" (UID: \"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a\") " pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:43:32.298862 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:32.298825    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a-metrics-certs\") pod \"network-metrics-daemon-fqmmn\" (UID: \"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a\") " pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:43:32.463218 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:32.463194    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lng7\""
Sep 17 16:43:32.471271 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:32.471251    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fqmmn"
Sep 17 16:43:32.774474 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:32.774303    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fqmmn"]
Sep 17 16:43:32.777080 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:43:32.777052    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ef9dbe_6ecf_42dc_9237_9bf1986b4d7a.slice/crio-fd4f96fe7f5a48bf67ec034722cb0c2cc24f6618086dca2641ef39e81d208cdf WatchSource:0}: Error finding container fd4f96fe7f5a48bf67ec034722cb0c2cc24f6618086dca2641ef39e81d208cdf: Status 404 returned error can't find the container with id fd4f96fe7f5a48bf67ec034722cb0c2cc24f6618086dca2641ef39e81d208cdf
Sep 17 16:43:33.029218 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:33.029140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fqmmn" event={"ID":"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a","Type":"ContainerStarted","Data":"fd4f96fe7f5a48bf67ec034722cb0c2cc24f6618086dca2641ef39e81d208cdf"}
Sep 17 16:43:33.239133 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:33.239110    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:35.037684 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:35.037648    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fqmmn" event={"ID":"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a","Type":"ContainerStarted","Data":"c80a5545143930d8dd44b42d7c311fbbac2e81e39782a7381992a0e342ef1f07"}
Sep 17 16:43:35.038120 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:35.037706    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fqmmn" event={"ID":"35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a","Type":"ContainerStarted","Data":"4e86ef9b21d08cc29382710bd036f75f452fabfbfa5c82447658a4c87c34ee42"}
Sep 17 16:43:35.054303 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:35.054261    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fqmmn" podStartSLOduration=131.653321324 podStartE2EDuration="2m13.054247022s" totalImagesPullingTime="1.400925698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:35.053945316 +0000 UTC m=+133.078853605" watchObservedRunningTime="2026-09-17 16:43:35.054247022 +0000 UTC m=+133.079155311"
Sep 17 16:43:39.325990 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.325922    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4" podUID="4512d740-dc14-4782-a5f6-67949e27ae41" containerName="registry" containerID="cri-o://6179a0dfc8132d41bcde27a03456ffb6ad8d377b05ce9b1ce7d8dc0383959bc9" gracePeriod=30
Sep 17 16:43:39.598560 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.598505    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:43:39.649253 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.649225    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4512d740-dc14-4782-a5f6-67949e27ae41-image-registry-private-configuration\") pod \"4512d740-dc14-4782-a5f6-67949e27ae41\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") "
Sep 17 16:43:39.649363 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.649266    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4512d740-dc14-4782-a5f6-67949e27ae41-ca-trust-extracted\") pod \"4512d740-dc14-4782-a5f6-67949e27ae41\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") "
Sep 17 16:43:39.649363 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.649303    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4512d740-dc14-4782-a5f6-67949e27ae41-registry-certificates\") pod \"4512d740-dc14-4782-a5f6-67949e27ae41\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") "
Sep 17 16:43:39.649446 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.649409    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4512d740-dc14-4782-a5f6-67949e27ae41-installation-pull-secrets\") pod \"4512d740-dc14-4782-a5f6-67949e27ae41\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") "
Sep 17 16:43:39.649498 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.649469    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4512d740-dc14-4782-a5f6-67949e27ae41-trusted-ca\") pod \"4512d740-dc14-4782-a5f6-67949e27ae41\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") "
Sep 17 16:43:39.649552 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.649497    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-bound-sa-token\") pod \"4512d740-dc14-4782-a5f6-67949e27ae41\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") "
Sep 17 16:43:39.649552 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.649526    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58t8v\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-kube-api-access-58t8v\") pod \"4512d740-dc14-4782-a5f6-67949e27ae41\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") "
Sep 17 16:43:39.649649 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.649569    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls\") pod \"4512d740-dc14-4782-a5f6-67949e27ae41\" (UID: \"4512d740-dc14-4782-a5f6-67949e27ae41\") "
Sep 17 16:43:39.649711 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.649654    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4512d740-dc14-4782-a5f6-67949e27ae41-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4512d740-dc14-4782-a5f6-67949e27ae41" (UID: "4512d740-dc14-4782-a5f6-67949e27ae41"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:43:39.649898 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.649846    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4512d740-dc14-4782-a5f6-67949e27ae41-registry-certificates\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:43:39.649898 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.649849    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4512d740-dc14-4782-a5f6-67949e27ae41-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4512d740-dc14-4782-a5f6-67949e27ae41" (UID: "4512d740-dc14-4782-a5f6-67949e27ae41"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:43:39.652177 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.652133    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4512d740-dc14-4782-a5f6-67949e27ae41-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4512d740-dc14-4782-a5f6-67949e27ae41" (UID: "4512d740-dc14-4782-a5f6-67949e27ae41"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:43:39.652290 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.652201    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4512d740-dc14-4782-a5f6-67949e27ae41" (UID: "4512d740-dc14-4782-a5f6-67949e27ae41"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:43:39.652290 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.652207    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-kube-api-access-58t8v" (OuterVolumeSpecName: "kube-api-access-58t8v") pod "4512d740-dc14-4782-a5f6-67949e27ae41" (UID: "4512d740-dc14-4782-a5f6-67949e27ae41"). InnerVolumeSpecName "kube-api-access-58t8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:43:39.652377 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.652288    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4512d740-dc14-4782-a5f6-67949e27ae41-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4512d740-dc14-4782-a5f6-67949e27ae41" (UID: "4512d740-dc14-4782-a5f6-67949e27ae41"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:43:39.652377 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.652313    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4512d740-dc14-4782-a5f6-67949e27ae41" (UID: "4512d740-dc14-4782-a5f6-67949e27ae41"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:43:39.657498 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.657476    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4512d740-dc14-4782-a5f6-67949e27ae41-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4512d740-dc14-4782-a5f6-67949e27ae41" (UID: "4512d740-dc14-4782-a5f6-67949e27ae41"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:43:39.750599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.750578    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4512d740-dc14-4782-a5f6-67949e27ae41-trusted-ca\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:43:39.750599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.750597    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-bound-sa-token\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:43:39.750713 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.750606    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58t8v\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-kube-api-access-58t8v\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:43:39.750713 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.750615    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4512d740-dc14-4782-a5f6-67949e27ae41-registry-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:43:39.750713 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.750624    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4512d740-dc14-4782-a5f6-67949e27ae41-image-registry-private-configuration\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:43:39.750713 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.750632    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4512d740-dc14-4782-a5f6-67949e27ae41-ca-trust-extracted\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:43:39.750713 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:39.750641    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4512d740-dc14-4782-a5f6-67949e27ae41-installation-pull-secrets\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:43:40.056994 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:40.056961    2577 generic.go:356] "Generic (PLEG): container finished" podID="4512d740-dc14-4782-a5f6-67949e27ae41" containerID="6179a0dfc8132d41bcde27a03456ffb6ad8d377b05ce9b1ce7d8dc0383959bc9" exitCode=0
Sep 17 16:43:40.057190 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:40.057003    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4"
Sep 17 16:43:40.057190 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:40.057048    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4" event={"ID":"4512d740-dc14-4782-a5f6-67949e27ae41","Type":"ContainerDied","Data":"6179a0dfc8132d41bcde27a03456ffb6ad8d377b05ce9b1ce7d8dc0383959bc9"}
Sep 17 16:43:40.057190 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:40.057108    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c7dc746d-zrqf4" event={"ID":"4512d740-dc14-4782-a5f6-67949e27ae41","Type":"ContainerDied","Data":"4a09d0c63e9e3d70e158ce50fbe7e7c666dc4d434791f4bb3ac9b8a88f015819"}
Sep 17 16:43:40.057190 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:40.057138    2577 scope.go:117] "RemoveContainer" containerID="6179a0dfc8132d41bcde27a03456ffb6ad8d377b05ce9b1ce7d8dc0383959bc9"
Sep 17 16:43:40.065826 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:40.065809    2577 scope.go:117] "RemoveContainer" containerID="6179a0dfc8132d41bcde27a03456ffb6ad8d377b05ce9b1ce7d8dc0383959bc9"
Sep 17 16:43:40.066123 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:43:40.066104    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6179a0dfc8132d41bcde27a03456ffb6ad8d377b05ce9b1ce7d8dc0383959bc9\": container with ID starting with 6179a0dfc8132d41bcde27a03456ffb6ad8d377b05ce9b1ce7d8dc0383959bc9 not found: ID does not exist" containerID="6179a0dfc8132d41bcde27a03456ffb6ad8d377b05ce9b1ce7d8dc0383959bc9"
Sep 17 16:43:40.066198 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:40.066131    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6179a0dfc8132d41bcde27a03456ffb6ad8d377b05ce9b1ce7d8dc0383959bc9"} err="failed to get container status \"6179a0dfc8132d41bcde27a03456ffb6ad8d377b05ce9b1ce7d8dc0383959bc9\": rpc error: code = NotFound desc = could not find container \"6179a0dfc8132d41bcde27a03456ffb6ad8d377b05ce9b1ce7d8dc0383959bc9\": container with ID starting with 6179a0dfc8132d41bcde27a03456ffb6ad8d377b05ce9b1ce7d8dc0383959bc9 not found: ID does not exist"
Sep 17 16:43:40.078755 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:40.078731    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c7dc746d-zrqf4"]
Sep 17 16:43:40.083025 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:40.083004    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c7dc746d-zrqf4"]
Sep 17 16:43:40.446635 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:40.446577    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4512d740-dc14-4782-a5f6-67949e27ae41" path="/var/lib/kubelet/pods/4512d740-dc14-4782-a5f6-67949e27ae41/volumes"
Sep 17 16:43:52.109831 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:52.109797    2577 generic.go:356] "Generic (PLEG): container finished" podID="e2fbafb9-cf8e-4b97-8971-c231a952ba1e" containerID="510a100c9b03884db42b9140b9b66b4ef81c6255df2b11499a58186084a099d8" exitCode=0
Sep 17 16:43:52.110305 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:52.109852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-678qx" event={"ID":"e2fbafb9-cf8e-4b97-8971-c231a952ba1e","Type":"ContainerDied","Data":"510a100c9b03884db42b9140b9b66b4ef81c6255df2b11499a58186084a099d8"}
Sep 17 16:43:52.110305 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:52.110125    2577 scope.go:117] "RemoveContainer" containerID="510a100c9b03884db42b9140b9b66b4ef81c6255df2b11499a58186084a099d8"
Sep 17 16:43:53.114879 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:53.114848    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-678qx" event={"ID":"e2fbafb9-cf8e-4b97-8971-c231a952ba1e","Type":"ContainerStarted","Data":"0a1f7c9246d3af58d001474bea04786daf9e8652ee6695ae200723fc0d2926bd"}
Sep 17 16:43:56.513124 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.513085    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74596f4558-4xsl9" podUID="a1d5c238-a754-4bc4-993b-362181e94ca3" containerName="console" containerID="cri-o://1c0868ef87827ff0f29d322e2a13a30a9a20f8ae51a5793056952dd9466b1db8" gracePeriod=15
Sep 17 16:43:56.780823 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.780769    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:56.878931 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.878900    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1d5c238-a754-4bc4-993b-362181e94ca3-console-serving-cert\") pod \"a1d5c238-a754-4bc4-993b-362181e94ca3\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") "
Sep 17 16:43:56.878931 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.878932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-oauth-serving-cert\") pod \"a1d5c238-a754-4bc4-993b-362181e94ca3\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") "
Sep 17 16:43:56.879225 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.878953    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-service-ca\") pod \"a1d5c238-a754-4bc4-993b-362181e94ca3\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") "
Sep 17 16:43:56.879225 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.878983    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdqq8\" (UniqueName: \"kubernetes.io/projected/a1d5c238-a754-4bc4-993b-362181e94ca3-kube-api-access-zdqq8\") pod \"a1d5c238-a754-4bc4-993b-362181e94ca3\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") "
Sep 17 16:43:56.879225 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.879010    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1d5c238-a754-4bc4-993b-362181e94ca3-console-oauth-config\") pod \"a1d5c238-a754-4bc4-993b-362181e94ca3\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") "
Sep 17 16:43:56.879225 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.879078    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-console-config\") pod \"a1d5c238-a754-4bc4-993b-362181e94ca3\" (UID: \"a1d5c238-a754-4bc4-993b-362181e94ca3\") "
Sep 17 16:43:56.879496 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.879468    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a1d5c238-a754-4bc4-993b-362181e94ca3" (UID: "a1d5c238-a754-4bc4-993b-362181e94ca3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:43:56.879496 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.879481    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-service-ca" (OuterVolumeSpecName: "service-ca") pod "a1d5c238-a754-4bc4-993b-362181e94ca3" (UID: "a1d5c238-a754-4bc4-993b-362181e94ca3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:43:56.879677 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.879545    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-console-config" (OuterVolumeSpecName: "console-config") pod "a1d5c238-a754-4bc4-993b-362181e94ca3" (UID: "a1d5c238-a754-4bc4-993b-362181e94ca3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:43:56.881255 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.881229    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d5c238-a754-4bc4-993b-362181e94ca3-kube-api-access-zdqq8" (OuterVolumeSpecName: "kube-api-access-zdqq8") pod "a1d5c238-a754-4bc4-993b-362181e94ca3" (UID: "a1d5c238-a754-4bc4-993b-362181e94ca3"). InnerVolumeSpecName "kube-api-access-zdqq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:43:56.881338 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.881296    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1d5c238-a754-4bc4-993b-362181e94ca3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a1d5c238-a754-4bc4-993b-362181e94ca3" (UID: "a1d5c238-a754-4bc4-993b-362181e94ca3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:43:56.881377 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.881339    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1d5c238-a754-4bc4-993b-362181e94ca3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a1d5c238-a754-4bc4-993b-362181e94ca3" (UID: "a1d5c238-a754-4bc4-993b-362181e94ca3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:43:56.979786 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.979762    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1d5c238-a754-4bc4-993b-362181e94ca3-console-serving-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:43:56.979786 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.979784    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-oauth-serving-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:43:56.979936 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.979793    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-service-ca\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:43:56.979936 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.979802    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdqq8\" (UniqueName: \"kubernetes.io/projected/a1d5c238-a754-4bc4-993b-362181e94ca3-kube-api-access-zdqq8\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:43:56.979936 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.979810    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1d5c238-a754-4bc4-993b-362181e94ca3-console-oauth-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:43:56.979936 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:56.979818    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1d5c238-a754-4bc4-993b-362181e94ca3-console-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:43:57.135563 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:57.135529    2577 generic.go:356] "Generic (PLEG): container finished" podID="a1d5c238-a754-4bc4-993b-362181e94ca3" containerID="1c0868ef87827ff0f29d322e2a13a30a9a20f8ae51a5793056952dd9466b1db8" exitCode=0
Sep 17 16:43:57.135728 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:57.135590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74596f4558-4xsl9" event={"ID":"a1d5c238-a754-4bc4-993b-362181e94ca3","Type":"ContainerDied","Data":"1c0868ef87827ff0f29d322e2a13a30a9a20f8ae51a5793056952dd9466b1db8"}
Sep 17 16:43:57.135728 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:57.135590    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74596f4558-4xsl9"
Sep 17 16:43:57.135728 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:57.135691    2577 scope.go:117] "RemoveContainer" containerID="1c0868ef87827ff0f29d322e2a13a30a9a20f8ae51a5793056952dd9466b1db8"
Sep 17 16:43:57.135877 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:57.135663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74596f4558-4xsl9" event={"ID":"a1d5c238-a754-4bc4-993b-362181e94ca3","Type":"ContainerDied","Data":"62981ea011e8289258506b1782af2e50c1c91973ec74f21041d7a393fd4849d5"}
Sep 17 16:43:57.144480 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:57.144465    2577 scope.go:117] "RemoveContainer" containerID="1c0868ef87827ff0f29d322e2a13a30a9a20f8ae51a5793056952dd9466b1db8"
Sep 17 16:43:57.144739 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:43:57.144717    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0868ef87827ff0f29d322e2a13a30a9a20f8ae51a5793056952dd9466b1db8\": container with ID starting with 1c0868ef87827ff0f29d322e2a13a30a9a20f8ae51a5793056952dd9466b1db8 not found: ID does not exist" containerID="1c0868ef87827ff0f29d322e2a13a30a9a20f8ae51a5793056952dd9466b1db8"
Sep 17 16:43:57.144821 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:57.144744    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0868ef87827ff0f29d322e2a13a30a9a20f8ae51a5793056952dd9466b1db8"} err="failed to get container status \"1c0868ef87827ff0f29d322e2a13a30a9a20f8ae51a5793056952dd9466b1db8\": rpc error: code = NotFound desc = could not find container \"1c0868ef87827ff0f29d322e2a13a30a9a20f8ae51a5793056952dd9466b1db8\": container with ID starting with 1c0868ef87827ff0f29d322e2a13a30a9a20f8ae51a5793056952dd9466b1db8 not found: ID does not exist"
Sep 17 16:43:57.157048 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:57.157026    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74596f4558-4xsl9"]
Sep 17 16:43:57.160745 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:57.160722    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74596f4558-4xsl9"]
Sep 17 16:43:58.446292 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:43:58.446263    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1d5c238-a754-4bc4-993b-362181e94ca3" path="/var/lib/kubelet/pods/a1d5c238-a754-4bc4-993b-362181e94ca3/volumes"
Sep 17 16:44:27.791706 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.791666    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-665c79c748-qvm8v"]
Sep 17 16:44:27.792140 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.791953    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d5c238-a754-4bc4-993b-362181e94ca3" containerName="console"
Sep 17 16:44:27.792140 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.791966    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d5c238-a754-4bc4-993b-362181e94ca3" containerName="console"
Sep 17 16:44:27.792140 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.791994    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4512d740-dc14-4782-a5f6-67949e27ae41" containerName="registry"
Sep 17 16:44:27.792140 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.792002    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4512d740-dc14-4782-a5f6-67949e27ae41" containerName="registry"
Sep 17 16:44:27.792140 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.792047    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1d5c238-a754-4bc4-993b-362181e94ca3" containerName="console"
Sep 17 16:44:27.792140 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.792056    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4512d740-dc14-4782-a5f6-67949e27ae41" containerName="registry"
Sep 17 16:44:27.796522 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.796499    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:27.799297 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.799269    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 16:44:27.800871 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.800846    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 16:44:27.801017 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.800890    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 16:44:27.801017 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.800927    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mbrl8\""
Sep 17 16:44:27.801017 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.800848    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 16:44:27.801253 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.801236    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 16:44:27.806191 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.806136    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 16:44:27.807841 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.807821    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-665c79c748-qvm8v"]
Sep 17 16:44:27.898022 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.897992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-config\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:27.898186 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.898025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-serving-cert\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:27.898186 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.898090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-oauth-config\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:27.898186 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.898122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98jmx\" (UniqueName: \"kubernetes.io/projected/0f070aa8-6a3b-46d5-a59a-40860413e53b-kube-api-access-98jmx\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:27.898321 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.898203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-service-ca\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:27.898321 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.898238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-oauth-serving-cert\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:27.898321 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.898257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-trusted-ca-bundle\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:27.999266 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.999240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-service-ca\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:27.999374 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.999271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-oauth-serving-cert\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:27.999374 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.999287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-trusted-ca-bundle\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:27.999374 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.999319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-config\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:27.999374 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.999337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-serving-cert\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:27.999374 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.999362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-oauth-config\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:27.999609 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:27.999377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98jmx\" (UniqueName: \"kubernetes.io/projected/0f070aa8-6a3b-46d5-a59a-40860413e53b-kube-api-access-98jmx\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:28.000038 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:28.000009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-oauth-serving-cert\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:28.000148 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:28.000019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-service-ca\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:28.000148 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:28.000049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-config\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:28.000396 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:28.000376    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-trusted-ca-bundle\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:28.001765 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:28.001743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-oauth-config\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:28.001993 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:28.001974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-serving-cert\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:28.006866 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:28.006841    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98jmx\" (UniqueName: \"kubernetes.io/projected/0f070aa8-6a3b-46d5-a59a-40860413e53b-kube-api-access-98jmx\") pod \"console-665c79c748-qvm8v\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") " pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:28.107025 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:28.106995    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:28.379044 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:28.378915    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-665c79c748-qvm8v"]
Sep 17 16:44:28.381829 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:44:28.381804    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f070aa8_6a3b_46d5_a59a_40860413e53b.slice/crio-ce8a3a799ef56e1460c81b530f2de4ced6e2208cd63ece56c95ad11f516f5fe6 WatchSource:0}: Error finding container ce8a3a799ef56e1460c81b530f2de4ced6e2208cd63ece56c95ad11f516f5fe6: Status 404 returned error can't find the container with id ce8a3a799ef56e1460c81b530f2de4ced6e2208cd63ece56c95ad11f516f5fe6
Sep 17 16:44:29.262303 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:29.262264    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-665c79c748-qvm8v" event={"ID":"0f070aa8-6a3b-46d5-a59a-40860413e53b","Type":"ContainerStarted","Data":"ac8d05ae49bf9054ecfab403ddc31ee050083a734e421cf5ee28e3e252f038ad"}
Sep 17 16:44:29.262665 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:29.262315    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-665c79c748-qvm8v" event={"ID":"0f070aa8-6a3b-46d5-a59a-40860413e53b","Type":"ContainerStarted","Data":"ce8a3a799ef56e1460c81b530f2de4ced6e2208cd63ece56c95ad11f516f5fe6"}
Sep 17 16:44:29.282372 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:29.282323    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-665c79c748-qvm8v" podStartSLOduration=2.282307102 podStartE2EDuration="2.282307102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:44:29.280507612 +0000 UTC m=+187.305415900" watchObservedRunningTime="2026-09-17 16:44:29.282307102 +0000 UTC m=+187.307215454"
Sep 17 16:44:38.108115 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:38.108070    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:38.108514 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:38.108136    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:38.112877 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:38.112853    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:44:38.299737 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:44:38.299706    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:46:00.094860 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:00.094823    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-665c79c748-qvm8v"]
Sep 17 16:46:22.383345 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:22.383318    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 16:46:22.388841 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:22.388811    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 16:46:22.394336 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:22.394313    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 16:46:25.115260 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.115199    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-665c79c748-qvm8v" podUID="0f070aa8-6a3b-46d5-a59a-40860413e53b" containerName="console" containerID="cri-o://ac8d05ae49bf9054ecfab403ddc31ee050083a734e421cf5ee28e3e252f038ad" gracePeriod=15
Sep 17 16:46:25.346144 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.346122    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:46:25.500053 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.500000    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-config\") pod \"0f070aa8-6a3b-46d5-a59a-40860413e53b\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") "
Sep 17 16:46:25.500053 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.500035    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98jmx\" (UniqueName: \"kubernetes.io/projected/0f070aa8-6a3b-46d5-a59a-40860413e53b-kube-api-access-98jmx\") pod \"0f070aa8-6a3b-46d5-a59a-40860413e53b\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") "
Sep 17 16:46:25.500219 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.500057    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-oauth-config\") pod \"0f070aa8-6a3b-46d5-a59a-40860413e53b\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") "
Sep 17 16:46:25.500219 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.500094    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-trusted-ca-bundle\") pod \"0f070aa8-6a3b-46d5-a59a-40860413e53b\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") "
Sep 17 16:46:25.500219 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.500153    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-serving-cert\") pod \"0f070aa8-6a3b-46d5-a59a-40860413e53b\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") "
Sep 17 16:46:25.500219 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.500197    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-oauth-serving-cert\") pod \"0f070aa8-6a3b-46d5-a59a-40860413e53b\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") "
Sep 17 16:46:25.500401 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.500228    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-service-ca\") pod \"0f070aa8-6a3b-46d5-a59a-40860413e53b\" (UID: \"0f070aa8-6a3b-46d5-a59a-40860413e53b\") "
Sep 17 16:46:25.500401 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.500379    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-config" (OuterVolumeSpecName: "console-config") pod "0f070aa8-6a3b-46d5-a59a-40860413e53b" (UID: "0f070aa8-6a3b-46d5-a59a-40860413e53b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:46:25.500637 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.500605    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0f070aa8-6a3b-46d5-a59a-40860413e53b" (UID: "0f070aa8-6a3b-46d5-a59a-40860413e53b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:46:25.500752 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.500650    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0f070aa8-6a3b-46d5-a59a-40860413e53b" (UID: "0f070aa8-6a3b-46d5-a59a-40860413e53b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:46:25.500752 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.500710    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-service-ca" (OuterVolumeSpecName: "service-ca") pod "0f070aa8-6a3b-46d5-a59a-40860413e53b" (UID: "0f070aa8-6a3b-46d5-a59a-40860413e53b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:46:25.502228 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.502199    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f070aa8-6a3b-46d5-a59a-40860413e53b-kube-api-access-98jmx" (OuterVolumeSpecName: "kube-api-access-98jmx") pod "0f070aa8-6a3b-46d5-a59a-40860413e53b" (UID: "0f070aa8-6a3b-46d5-a59a-40860413e53b"). InnerVolumeSpecName "kube-api-access-98jmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:46:25.502310 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.502226    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0f070aa8-6a3b-46d5-a59a-40860413e53b" (UID: "0f070aa8-6a3b-46d5-a59a-40860413e53b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:46:25.502310 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.502252    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0f070aa8-6a3b-46d5-a59a-40860413e53b" (UID: "0f070aa8-6a3b-46d5-a59a-40860413e53b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:46:25.600665 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.600641    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-oauth-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:46:25.600665 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.600664    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-trusted-ca-bundle\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:46:25.600798 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.600677    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-serving-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:46:25.600798 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.600690    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-oauth-serving-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:46:25.600798 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.600703    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-service-ca\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:46:25.600798 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.600714    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f070aa8-6a3b-46d5-a59a-40860413e53b-console-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:46:25.600798 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.600726    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98jmx\" (UniqueName: \"kubernetes.io/projected/0f070aa8-6a3b-46d5-a59a-40860413e53b-kube-api-access-98jmx\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:46:25.689367 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.689334    2577 generic.go:356] "Generic (PLEG): container finished" podID="0f070aa8-6a3b-46d5-a59a-40860413e53b" containerID="ac8d05ae49bf9054ecfab403ddc31ee050083a734e421cf5ee28e3e252f038ad" exitCode=0
Sep 17 16:46:25.689478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.689385    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-665c79c748-qvm8v"
Sep 17 16:46:25.689478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.689392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-665c79c748-qvm8v" event={"ID":"0f070aa8-6a3b-46d5-a59a-40860413e53b","Type":"ContainerDied","Data":"ac8d05ae49bf9054ecfab403ddc31ee050083a734e421cf5ee28e3e252f038ad"}
Sep 17 16:46:25.689478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.689437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-665c79c748-qvm8v" event={"ID":"0f070aa8-6a3b-46d5-a59a-40860413e53b","Type":"ContainerDied","Data":"ce8a3a799ef56e1460c81b530f2de4ced6e2208cd63ece56c95ad11f516f5fe6"}
Sep 17 16:46:25.689478 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.689460    2577 scope.go:117] "RemoveContainer" containerID="ac8d05ae49bf9054ecfab403ddc31ee050083a734e421cf5ee28e3e252f038ad"
Sep 17 16:46:25.698026 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.698003    2577 scope.go:117] "RemoveContainer" containerID="ac8d05ae49bf9054ecfab403ddc31ee050083a734e421cf5ee28e3e252f038ad"
Sep 17 16:46:25.698351 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:46:25.698334    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac8d05ae49bf9054ecfab403ddc31ee050083a734e421cf5ee28e3e252f038ad\": container with ID starting with ac8d05ae49bf9054ecfab403ddc31ee050083a734e421cf5ee28e3e252f038ad not found: ID does not exist" containerID="ac8d05ae49bf9054ecfab403ddc31ee050083a734e421cf5ee28e3e252f038ad"
Sep 17 16:46:25.698419 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.698358    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac8d05ae49bf9054ecfab403ddc31ee050083a734e421cf5ee28e3e252f038ad"} err="failed to get container status \"ac8d05ae49bf9054ecfab403ddc31ee050083a734e421cf5ee28e3e252f038ad\": rpc error: code = NotFound desc = could not find container \"ac8d05ae49bf9054ecfab403ddc31ee050083a734e421cf5ee28e3e252f038ad\": container with ID starting with ac8d05ae49bf9054ecfab403ddc31ee050083a734e421cf5ee28e3e252f038ad not found: ID does not exist"
Sep 17 16:46:25.711376 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.711354    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-665c79c748-qvm8v"]
Sep 17 16:46:25.714774 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:25.714752    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-665c79c748-qvm8v"]
Sep 17 16:46:26.446554 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:26.446518    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f070aa8-6a3b-46d5-a59a-40860413e53b" path="/var/lib/kubelet/pods/0f070aa8-6a3b-46d5-a59a-40860413e53b/volumes"
Sep 17 16:46:50.116204 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.116170    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"]
Sep 17 16:46:50.116687 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.116458    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f070aa8-6a3b-46d5-a59a-40860413e53b" containerName="console"
Sep 17 16:46:50.116687 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.116469    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f070aa8-6a3b-46d5-a59a-40860413e53b" containerName="console"
Sep 17 16:46:50.116687 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.116534    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f070aa8-6a3b-46d5-a59a-40860413e53b" containerName="console"
Sep 17 16:46:50.119527 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.119510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"
Sep 17 16:46:50.122491 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.122458    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v9bj2\""
Sep 17 16:46:50.122616 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.122538    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 16:46:50.122616 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.122559    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 16:46:50.129114 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.129093    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"]
Sep 17 16:46:50.270508 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.270475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d674f3b5-1be7-436f-b79b-5aa940d05f1b-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv\" (UID: \"d674f3b5-1be7-436f-b79b-5aa940d05f1b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"
Sep 17 16:46:50.270655 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.270517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d674f3b5-1be7-436f-b79b-5aa940d05f1b-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv\" (UID: \"d674f3b5-1be7-436f-b79b-5aa940d05f1b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"
Sep 17 16:46:50.270655 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.270593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mllr2\" (UniqueName: \"kubernetes.io/projected/d674f3b5-1be7-436f-b79b-5aa940d05f1b-kube-api-access-mllr2\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv\" (UID: \"d674f3b5-1be7-436f-b79b-5aa940d05f1b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"
Sep 17 16:46:50.371091 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.371016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d674f3b5-1be7-436f-b79b-5aa940d05f1b-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv\" (UID: \"d674f3b5-1be7-436f-b79b-5aa940d05f1b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"
Sep 17 16:46:50.371091 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.371062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d674f3b5-1be7-436f-b79b-5aa940d05f1b-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv\" (UID: \"d674f3b5-1be7-436f-b79b-5aa940d05f1b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"
Sep 17 16:46:50.371282 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.371170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mllr2\" (UniqueName: \"kubernetes.io/projected/d674f3b5-1be7-436f-b79b-5aa940d05f1b-kube-api-access-mllr2\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv\" (UID: \"d674f3b5-1be7-436f-b79b-5aa940d05f1b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"
Sep 17 16:46:50.371418 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.371400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d674f3b5-1be7-436f-b79b-5aa940d05f1b-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv\" (UID: \"d674f3b5-1be7-436f-b79b-5aa940d05f1b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"
Sep 17 16:46:50.371465 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.371448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d674f3b5-1be7-436f-b79b-5aa940d05f1b-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv\" (UID: \"d674f3b5-1be7-436f-b79b-5aa940d05f1b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"
Sep 17 16:46:50.380572 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.380549    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mllr2\" (UniqueName: \"kubernetes.io/projected/d674f3b5-1be7-436f-b79b-5aa940d05f1b-kube-api-access-mllr2\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv\" (UID: \"d674f3b5-1be7-436f-b79b-5aa940d05f1b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"
Sep 17 16:46:50.428955 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.428922    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"
Sep 17 16:46:50.699319 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.699268    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"]
Sep 17 16:46:50.701534 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:46:50.701507    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd674f3b5_1be7_436f_b79b_5aa940d05f1b.slice/crio-13dfab1412c1e7c633278aaad9c4eb576f8064023adf48b0a441d482528fecc5 WatchSource:0}: Error finding container 13dfab1412c1e7c633278aaad9c4eb576f8064023adf48b0a441d482528fecc5: Status 404 returned error can't find the container with id 13dfab1412c1e7c633278aaad9c4eb576f8064023adf48b0a441d482528fecc5
Sep 17 16:46:50.703931 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.703912    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:46:50.788578 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:50.788546    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv" event={"ID":"d674f3b5-1be7-436f-b79b-5aa940d05f1b","Type":"ContainerStarted","Data":"13dfab1412c1e7c633278aaad9c4eb576f8064023adf48b0a441d482528fecc5"}
Sep 17 16:46:57.824276 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:57.824243    2577 generic.go:356] "Generic (PLEG): container finished" podID="d674f3b5-1be7-436f-b79b-5aa940d05f1b" containerID="44d2920e61f9bb8d406d55d748914afdee07e1c073fd867f36b8909e7fdeda43" exitCode=0
Sep 17 16:46:57.824682 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:46:57.824331    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv" event={"ID":"d674f3b5-1be7-436f-b79b-5aa940d05f1b","Type":"ContainerDied","Data":"44d2920e61f9bb8d406d55d748914afdee07e1c073fd867f36b8909e7fdeda43"}
Sep 17 16:47:00.838516 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:00.838481    2577 generic.go:356] "Generic (PLEG): container finished" podID="d674f3b5-1be7-436f-b79b-5aa940d05f1b" containerID="89c53530b7231feeb6b17f33fcee78538785cc85f98dfa3afba4ae15cd43ddc9" exitCode=0
Sep 17 16:47:00.838875 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:00.838550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv" event={"ID":"d674f3b5-1be7-436f-b79b-5aa940d05f1b","Type":"ContainerDied","Data":"89c53530b7231feeb6b17f33fcee78538785cc85f98dfa3afba4ae15cd43ddc9"}
Sep 17 16:47:08.874738 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:08.874703    2577 generic.go:356] "Generic (PLEG): container finished" podID="d674f3b5-1be7-436f-b79b-5aa940d05f1b" containerID="cc1a5fcff6020b725211be55cbf856f601f7c35adbad859114af90389bd59a43" exitCode=0
Sep 17 16:47:08.875108 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:08.874761    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv" event={"ID":"d674f3b5-1be7-436f-b79b-5aa940d05f1b","Type":"ContainerDied","Data":"cc1a5fcff6020b725211be55cbf856f601f7c35adbad859114af90389bd59a43"}
Sep 17 16:47:10.011582 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:10.011553    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"
Sep 17 16:47:10.024624 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:10.024601    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mllr2\" (UniqueName: \"kubernetes.io/projected/d674f3b5-1be7-436f-b79b-5aa940d05f1b-kube-api-access-mllr2\") pod \"d674f3b5-1be7-436f-b79b-5aa940d05f1b\" (UID: \"d674f3b5-1be7-436f-b79b-5aa940d05f1b\") "
Sep 17 16:47:10.024716 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:10.024656    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d674f3b5-1be7-436f-b79b-5aa940d05f1b-bundle\") pod \"d674f3b5-1be7-436f-b79b-5aa940d05f1b\" (UID: \"d674f3b5-1be7-436f-b79b-5aa940d05f1b\") "
Sep 17 16:47:10.024716 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:10.024674    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d674f3b5-1be7-436f-b79b-5aa940d05f1b-util\") pod \"d674f3b5-1be7-436f-b79b-5aa940d05f1b\" (UID: \"d674f3b5-1be7-436f-b79b-5aa940d05f1b\") "
Sep 17 16:47:10.025342 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:10.025318    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d674f3b5-1be7-436f-b79b-5aa940d05f1b-bundle" (OuterVolumeSpecName: "bundle") pod "d674f3b5-1be7-436f-b79b-5aa940d05f1b" (UID: "d674f3b5-1be7-436f-b79b-5aa940d05f1b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:47:10.026769 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:10.026740    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d674f3b5-1be7-436f-b79b-5aa940d05f1b-kube-api-access-mllr2" (OuterVolumeSpecName: "kube-api-access-mllr2") pod "d674f3b5-1be7-436f-b79b-5aa940d05f1b" (UID: "d674f3b5-1be7-436f-b79b-5aa940d05f1b"). InnerVolumeSpecName "kube-api-access-mllr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:47:10.028929 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:10.028903    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d674f3b5-1be7-436f-b79b-5aa940d05f1b-util" (OuterVolumeSpecName: "util") pod "d674f3b5-1be7-436f-b79b-5aa940d05f1b" (UID: "d674f3b5-1be7-436f-b79b-5aa940d05f1b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:47:10.125203 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:10.125181    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d674f3b5-1be7-436f-b79b-5aa940d05f1b-bundle\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:47:10.125203 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:10.125201    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d674f3b5-1be7-436f-b79b-5aa940d05f1b-util\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:47:10.125322 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:10.125210    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mllr2\" (UniqueName: \"kubernetes.io/projected/d674f3b5-1be7-436f-b79b-5aa940d05f1b-kube-api-access-mllr2\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:47:10.884084 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:10.884058    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv"
Sep 17 16:47:10.884259 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:10.884082    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1atpspv" event={"ID":"d674f3b5-1be7-436f-b79b-5aa940d05f1b","Type":"ContainerDied","Data":"13dfab1412c1e7c633278aaad9c4eb576f8064023adf48b0a441d482528fecc5"}
Sep 17 16:47:10.884259 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:10.884127    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13dfab1412c1e7c633278aaad9c4eb576f8064023adf48b0a441d482528fecc5"
Sep 17 16:47:17.519746 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.519655    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9"]
Sep 17 16:47:17.520271 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.519997    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d674f3b5-1be7-436f-b79b-5aa940d05f1b" containerName="util"
Sep 17 16:47:17.520271 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.520014    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d674f3b5-1be7-436f-b79b-5aa940d05f1b" containerName="util"
Sep 17 16:47:17.520271 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.520027    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d674f3b5-1be7-436f-b79b-5aa940d05f1b" containerName="pull"
Sep 17 16:47:17.520271 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.520035    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d674f3b5-1be7-436f-b79b-5aa940d05f1b" containerName="pull"
Sep 17 16:47:17.520271 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.520045    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d674f3b5-1be7-436f-b79b-5aa940d05f1b" containerName="extract"
Sep 17 16:47:17.520271 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.520052    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d674f3b5-1be7-436f-b79b-5aa940d05f1b" containerName="extract"
Sep 17 16:47:17.520271 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.520106    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d674f3b5-1be7-436f-b79b-5aa940d05f1b" containerName="extract"
Sep 17 16:47:17.522690 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.522670    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9"
Sep 17 16:47:17.525777 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.525750    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 16:47:17.525953 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.525829    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-qfslv\""
Sep 17 16:47:17.526044 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.525968    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 16:47:17.526044 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.526013    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 16:47:17.537917 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.537892    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9"]
Sep 17 16:47:17.570317 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.570295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d1a8bf07-94ca-4551-b6ea-2a97d700ed1a-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tddb9\" (UID: \"d1a8bf07-94ca-4551-b6ea-2a97d700ed1a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9"
Sep 17 16:47:17.570440 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.570358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4zld\" (UniqueName: \"kubernetes.io/projected/d1a8bf07-94ca-4551-b6ea-2a97d700ed1a-kube-api-access-j4zld\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tddb9\" (UID: \"d1a8bf07-94ca-4551-b6ea-2a97d700ed1a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9"
Sep 17 16:47:17.670809 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.670778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d1a8bf07-94ca-4551-b6ea-2a97d700ed1a-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tddb9\" (UID: \"d1a8bf07-94ca-4551-b6ea-2a97d700ed1a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9"
Sep 17 16:47:17.670924 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.670826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4zld\" (UniqueName: \"kubernetes.io/projected/d1a8bf07-94ca-4551-b6ea-2a97d700ed1a-kube-api-access-j4zld\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tddb9\" (UID: \"d1a8bf07-94ca-4551-b6ea-2a97d700ed1a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9"
Sep 17 16:47:17.673336 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.673313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d1a8bf07-94ca-4551-b6ea-2a97d700ed1a-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tddb9\" (UID: \"d1a8bf07-94ca-4551-b6ea-2a97d700ed1a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9"
Sep 17 16:47:17.680051 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.680018    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4zld\" (UniqueName: \"kubernetes.io/projected/d1a8bf07-94ca-4551-b6ea-2a97d700ed1a-kube-api-access-j4zld\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tddb9\" (UID: \"d1a8bf07-94ca-4551-b6ea-2a97d700ed1a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9"
Sep 17 16:47:17.832884 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:17.832858    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9"
Sep 17 16:47:18.114913 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:18.114885    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9"]
Sep 17 16:47:18.116347 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:47:18.116317    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a8bf07_94ca_4551_b6ea_2a97d700ed1a.slice/crio-d720cabbda472529a6ea9cffb5839acd9860d779b49814e5fd246e3f0c61e1b6 WatchSource:0}: Error finding container d720cabbda472529a6ea9cffb5839acd9860d779b49814e5fd246e3f0c61e1b6: Status 404 returned error can't find the container with id d720cabbda472529a6ea9cffb5839acd9860d779b49814e5fd246e3f0c61e1b6
Sep 17 16:47:18.917980 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:18.917947    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9" event={"ID":"d1a8bf07-94ca-4551-b6ea-2a97d700ed1a","Type":"ContainerStarted","Data":"d720cabbda472529a6ea9cffb5839acd9860d779b49814e5fd246e3f0c61e1b6"}
Sep 17 16:47:21.933652 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:21.933614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9" event={"ID":"d1a8bf07-94ca-4551-b6ea-2a97d700ed1a","Type":"ContainerStarted","Data":"6ca650c5d9b6f2637a60ff1076555bb3718547080e9f910771d6dae3c06eaac4"}
Sep 17 16:47:21.934030 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:21.933724    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9"
Sep 17 16:47:21.987143 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:21.987095    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9" podStartSLOduration=2.039151099 podStartE2EDuration="4.98708357s" totalImagesPullingTime="2.947932471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:47:21.986099839 +0000 UTC m=+360.011008127" watchObservedRunningTime="2026-09-17 16:47:21.98708357 +0000 UTC m=+360.011991858"
Sep 17 16:47:22.069019 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.068987    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5k5k2"]
Sep 17 16:47:22.072195 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.072153    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-5k5k2"
Sep 17 16:47:22.074928 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.074903    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 16:47:22.075034 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.074930    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 16:47:22.075195 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.075181    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-p8b27\""
Sep 17 16:47:22.082182 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.082147    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5k5k2"]
Sep 17 16:47:22.104312 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.104286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f273da78-c7b2-41b6-b71c-6c3f1c16c463-cabundle0\") pod \"keda-operator-579c8958f7-5k5k2\" (UID: \"f273da78-c7b2-41b6-b71c-6c3f1c16c463\") " pod="openshift-keda/keda-operator-579c8958f7-5k5k2"
Sep 17 16:47:22.104408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.104364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6bgd\" (UniqueName: \"kubernetes.io/projected/f273da78-c7b2-41b6-b71c-6c3f1c16c463-kube-api-access-j6bgd\") pod \"keda-operator-579c8958f7-5k5k2\" (UID: \"f273da78-c7b2-41b6-b71c-6c3f1c16c463\") " pod="openshift-keda/keda-operator-579c8958f7-5k5k2"
Sep 17 16:47:22.104408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.104399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f273da78-c7b2-41b6-b71c-6c3f1c16c463-certificates\") pod \"keda-operator-579c8958f7-5k5k2\" (UID: \"f273da78-c7b2-41b6-b71c-6c3f1c16c463\") " pod="openshift-keda/keda-operator-579c8958f7-5k5k2"
Sep 17 16:47:22.205707 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.205654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f273da78-c7b2-41b6-b71c-6c3f1c16c463-cabundle0\") pod \"keda-operator-579c8958f7-5k5k2\" (UID: \"f273da78-c7b2-41b6-b71c-6c3f1c16c463\") " pod="openshift-keda/keda-operator-579c8958f7-5k5k2"
Sep 17 16:47:22.205707 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.205704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6bgd\" (UniqueName: \"kubernetes.io/projected/f273da78-c7b2-41b6-b71c-6c3f1c16c463-kube-api-access-j6bgd\") pod \"keda-operator-579c8958f7-5k5k2\" (UID: \"f273da78-c7b2-41b6-b71c-6c3f1c16c463\") " pod="openshift-keda/keda-operator-579c8958f7-5k5k2"
Sep 17 16:47:22.205860 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.205725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f273da78-c7b2-41b6-b71c-6c3f1c16c463-certificates\") pod \"keda-operator-579c8958f7-5k5k2\" (UID: \"f273da78-c7b2-41b6-b71c-6c3f1c16c463\") " pod="openshift-keda/keda-operator-579c8958f7-5k5k2"
Sep 17 16:47:22.205860 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:47:22.205814    2577 secret.go:281] references non-existent secret key: ca.crt
Sep 17 16:47:22.205860 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:47:22.205827    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 16:47:22.205860 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:47:22.205835    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-5k5k2: references non-existent secret key: ca.crt
Sep 17 16:47:22.206143 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:47:22.205882    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f273da78-c7b2-41b6-b71c-6c3f1c16c463-certificates podName:f273da78-c7b2-41b6-b71c-6c3f1c16c463 nodeName:}" failed. No retries permitted until 2026-09-17 16:47:22.705866475 +0000 UTC m=+360.730774743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f273da78-c7b2-41b6-b71c-6c3f1c16c463-certificates") pod "keda-operator-579c8958f7-5k5k2" (UID: "f273da78-c7b2-41b6-b71c-6c3f1c16c463") : references non-existent secret key: ca.crt
Sep 17 16:47:22.206408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.206386    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f273da78-c7b2-41b6-b71c-6c3f1c16c463-cabundle0\") pod \"keda-operator-579c8958f7-5k5k2\" (UID: \"f273da78-c7b2-41b6-b71c-6c3f1c16c463\") " pod="openshift-keda/keda-operator-579c8958f7-5k5k2"
Sep 17 16:47:22.215377 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.215354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6bgd\" (UniqueName: \"kubernetes.io/projected/f273da78-c7b2-41b6-b71c-6c3f1c16c463-kube-api-access-j6bgd\") pod \"keda-operator-579c8958f7-5k5k2\" (UID: \"f273da78-c7b2-41b6-b71c-6c3f1c16c463\") " pod="openshift-keda/keda-operator-579c8958f7-5k5k2"
Sep 17 16:47:22.674225 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.674193    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nhvz2"]
Sep 17 16:47:22.678587 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.678570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-nhvz2"
Sep 17 16:47:22.681300 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.681277    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 16:47:22.691953 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.691931    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nhvz2"]
Sep 17 16:47:22.708977 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.708949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrtn\" (UniqueName: \"kubernetes.io/projected/83cc3865-385a-440f-a2f6-8978a5ee0b1b-kube-api-access-rwrtn\") pod \"keda-admission-6c4d864b86-nhvz2\" (UID: \"83cc3865-385a-440f-a2f6-8978a5ee0b1b\") " pod="openshift-keda/keda-admission-6c4d864b86-nhvz2"
Sep 17 16:47:22.709096 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.708994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f273da78-c7b2-41b6-b71c-6c3f1c16c463-certificates\") pod \"keda-operator-579c8958f7-5k5k2\" (UID: \"f273da78-c7b2-41b6-b71c-6c3f1c16c463\") " pod="openshift-keda/keda-operator-579c8958f7-5k5k2"
Sep 17 16:47:22.709096 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.709013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/83cc3865-385a-440f-a2f6-8978a5ee0b1b-certificates\") pod \"keda-admission-6c4d864b86-nhvz2\" (UID: \"83cc3865-385a-440f-a2f6-8978a5ee0b1b\") " pod="openshift-keda/keda-admission-6c4d864b86-nhvz2"
Sep 17 16:47:22.711312 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.711294    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f273da78-c7b2-41b6-b71c-6c3f1c16c463-certificates\") pod \"keda-operator-579c8958f7-5k5k2\" (UID: \"f273da78-c7b2-41b6-b71c-6c3f1c16c463\") " pod="openshift-keda/keda-operator-579c8958f7-5k5k2"
Sep 17 16:47:22.810122 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.810091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrtn\" (UniqueName: \"kubernetes.io/projected/83cc3865-385a-440f-a2f6-8978a5ee0b1b-kube-api-access-rwrtn\") pod \"keda-admission-6c4d864b86-nhvz2\" (UID: \"83cc3865-385a-440f-a2f6-8978a5ee0b1b\") " pod="openshift-keda/keda-admission-6c4d864b86-nhvz2"
Sep 17 16:47:22.810262 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.810146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/83cc3865-385a-440f-a2f6-8978a5ee0b1b-certificates\") pod \"keda-admission-6c4d864b86-nhvz2\" (UID: \"83cc3865-385a-440f-a2f6-8978a5ee0b1b\") " pod="openshift-keda/keda-admission-6c4d864b86-nhvz2"
Sep 17 16:47:22.812438 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.812409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/83cc3865-385a-440f-a2f6-8978a5ee0b1b-certificates\") pod \"keda-admission-6c4d864b86-nhvz2\" (UID: \"83cc3865-385a-440f-a2f6-8978a5ee0b1b\") " pod="openshift-keda/keda-admission-6c4d864b86-nhvz2"
Sep 17 16:47:22.822629 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.822608    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrtn\" (UniqueName: \"kubernetes.io/projected/83cc3865-385a-440f-a2f6-8978a5ee0b1b-kube-api-access-rwrtn\") pod \"keda-admission-6c4d864b86-nhvz2\" (UID: \"83cc3865-385a-440f-a2f6-8978a5ee0b1b\") " pod="openshift-keda/keda-admission-6c4d864b86-nhvz2"
Sep 17 16:47:22.985672 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.985618    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-p8b27\""
Sep 17 16:47:22.987693 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.987676    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-nhvz2"
Sep 17 16:47:22.993511 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:22.993497    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-5k5k2"
Sep 17 16:47:23.262293 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:23.262222    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nhvz2"]
Sep 17 16:47:23.264924 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:23.264902    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5k5k2"]
Sep 17 16:47:23.265705 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:47:23.265676    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83cc3865_385a_440f_a2f6_8978a5ee0b1b.slice/crio-110b063c6911a013dcbc2f1f9df491111f790b26f7871c453b02c253cd1e2a1b WatchSource:0}: Error finding container 110b063c6911a013dcbc2f1f9df491111f790b26f7871c453b02c253cd1e2a1b: Status 404 returned error can't find the container with id 110b063c6911a013dcbc2f1f9df491111f790b26f7871c453b02c253cd1e2a1b
Sep 17 16:47:23.267244 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:47:23.267222    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf273da78_c7b2_41b6_b71c_6c3f1c16c463.slice/crio-61cb3608baa36cf8947281258626e6f6253df9634ff3dd60e5c64eea7713ff33 WatchSource:0}: Error finding container 61cb3608baa36cf8947281258626e6f6253df9634ff3dd60e5c64eea7713ff33: Status 404 returned error can't find the container with id 61cb3608baa36cf8947281258626e6f6253df9634ff3dd60e5c64eea7713ff33
Sep 17 16:47:23.943761 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:23.943724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-nhvz2" event={"ID":"83cc3865-385a-440f-a2f6-8978a5ee0b1b","Type":"ContainerStarted","Data":"110b063c6911a013dcbc2f1f9df491111f790b26f7871c453b02c253cd1e2a1b"}
Sep 17 16:47:23.945038 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:23.945007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-5k5k2" event={"ID":"f273da78-c7b2-41b6-b71c-6c3f1c16c463","Type":"ContainerStarted","Data":"61cb3608baa36cf8947281258626e6f6253df9634ff3dd60e5c64eea7713ff33"}
Sep 17 16:47:24.951911 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:24.951869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-nhvz2" event={"ID":"83cc3865-385a-440f-a2f6-8978a5ee0b1b","Type":"ContainerStarted","Data":"90b5fe071d3956bcbaaeac2ed566b1a784267bcaba3c0a36f7ec133bf09f1ada"}
Sep 17 16:47:24.952350 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:24.952104    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-nhvz2"
Sep 17 16:47:24.969753 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:24.969708    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-nhvz2" podStartSLOduration=1.481691485 podStartE2EDuration="2.969693716s" totalImagesPullingTime="1.488002231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:47:24.968724197 +0000 UTC m=+362.993632487" watchObservedRunningTime="2026-09-17 16:47:24.969693716 +0000 UTC m=+362.994602004"
Sep 17 16:47:26.962636 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:26.962553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-5k5k2" event={"ID":"f273da78-c7b2-41b6-b71c-6c3f1c16c463","Type":"ContainerStarted","Data":"268a41650d3ac76af265824f544586c548f004f6ee76fda7d7957dcd7b04c599"}
Sep 17 16:47:26.962636 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:26.962629    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-5k5k2"
Sep 17 16:47:26.981902 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:26.981858    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-5k5k2" podStartSLOduration=1.6967132409999999 podStartE2EDuration="4.981841957s" totalImagesPullingTime="3.285128716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:47:26.979934893 +0000 UTC m=+365.004843182" watchObservedRunningTime="2026-09-17 16:47:26.981841957 +0000 UTC m=+365.006750244"
Sep 17 16:47:42.940150 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:42.940110    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tddb9"
Sep 17 16:47:45.959827 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:45.959793    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-nhvz2"
Sep 17 16:47:47.969931 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:47:47.969896    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-5k5k2"
Sep 17 16:48:41.427945 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.427912    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz"]
Sep 17 16:48:41.431143 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.431126    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz"
Sep 17 16:48:41.433618 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.433595    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 16:48:41.433750 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.433681    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-5pzrn\""
Sep 17 16:48:41.434832 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.434813    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 16:48:41.434963 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.434815    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 16:48:41.440097 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.440067    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz"]
Sep 17 16:48:41.467352 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.467324    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-2qzhg"]
Sep 17 16:48:41.470293 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.470278    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2qzhg"
Sep 17 16:48:41.472805 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.472786    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 16:48:41.472901 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.472879    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-42vlc\""
Sep 17 16:48:41.478359 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.478339    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2qzhg"]
Sep 17 16:48:41.584041 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.584019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6211f1f7-b0dc-4d47-bef4-f51028d720ad-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-7mlwz\" (UID: \"6211f1f7-b0dc-4d47-bef4-f51028d720ad\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz"
Sep 17 16:48:41.584182 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.584062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gc2z\" (UniqueName: \"kubernetes.io/projected/c45a52e9-4d7d-464e-9ae6-9e73113e5585-kube-api-access-4gc2z\") pod \"seaweedfs-64568bcd49-2qzhg\" (UID: \"c45a52e9-4d7d-464e-9ae6-9e73113e5585\") " pod="kserve/seaweedfs-64568bcd49-2qzhg"
Sep 17 16:48:41.584257 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.584187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shkmw\" (UniqueName: \"kubernetes.io/projected/6211f1f7-b0dc-4d47-bef4-f51028d720ad-kube-api-access-shkmw\") pod \"llmisvc-controller-manager-645f9c8fd6-7mlwz\" (UID: \"6211f1f7-b0dc-4d47-bef4-f51028d720ad\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz"
Sep 17 16:48:41.584315 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.584248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c45a52e9-4d7d-464e-9ae6-9e73113e5585-data\") pod \"seaweedfs-64568bcd49-2qzhg\" (UID: \"c45a52e9-4d7d-464e-9ae6-9e73113e5585\") " pod="kserve/seaweedfs-64568bcd49-2qzhg"
Sep 17 16:48:41.685447 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.685391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gc2z\" (UniqueName: \"kubernetes.io/projected/c45a52e9-4d7d-464e-9ae6-9e73113e5585-kube-api-access-4gc2z\") pod \"seaweedfs-64568bcd49-2qzhg\" (UID: \"c45a52e9-4d7d-464e-9ae6-9e73113e5585\") " pod="kserve/seaweedfs-64568bcd49-2qzhg"
Sep 17 16:48:41.685447 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.685433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shkmw\" (UniqueName: \"kubernetes.io/projected/6211f1f7-b0dc-4d47-bef4-f51028d720ad-kube-api-access-shkmw\") pod \"llmisvc-controller-manager-645f9c8fd6-7mlwz\" (UID: \"6211f1f7-b0dc-4d47-bef4-f51028d720ad\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz"
Sep 17 16:48:41.685613 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.685455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c45a52e9-4d7d-464e-9ae6-9e73113e5585-data\") pod \"seaweedfs-64568bcd49-2qzhg\" (UID: \"c45a52e9-4d7d-464e-9ae6-9e73113e5585\") " pod="kserve/seaweedfs-64568bcd49-2qzhg"
Sep 17 16:48:41.685613 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.685493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6211f1f7-b0dc-4d47-bef4-f51028d720ad-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-7mlwz\" (UID: \"6211f1f7-b0dc-4d47-bef4-f51028d720ad\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz"
Sep 17 16:48:41.685940 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.685922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c45a52e9-4d7d-464e-9ae6-9e73113e5585-data\") pod \"seaweedfs-64568bcd49-2qzhg\" (UID: \"c45a52e9-4d7d-464e-9ae6-9e73113e5585\") " pod="kserve/seaweedfs-64568bcd49-2qzhg"
Sep 17 16:48:41.687872 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.687849    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6211f1f7-b0dc-4d47-bef4-f51028d720ad-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-7mlwz\" (UID: \"6211f1f7-b0dc-4d47-bef4-f51028d720ad\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz"
Sep 17 16:48:41.693589 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.693562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shkmw\" (UniqueName: \"kubernetes.io/projected/6211f1f7-b0dc-4d47-bef4-f51028d720ad-kube-api-access-shkmw\") pod \"llmisvc-controller-manager-645f9c8fd6-7mlwz\" (UID: \"6211f1f7-b0dc-4d47-bef4-f51028d720ad\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz"
Sep 17 16:48:41.694398 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.694376    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gc2z\" (UniqueName: \"kubernetes.io/projected/c45a52e9-4d7d-464e-9ae6-9e73113e5585-kube-api-access-4gc2z\") pod \"seaweedfs-64568bcd49-2qzhg\" (UID: \"c45a52e9-4d7d-464e-9ae6-9e73113e5585\") " pod="kserve/seaweedfs-64568bcd49-2qzhg"
Sep 17 16:48:41.741275 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.741255    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz"
Sep 17 16:48:41.778817 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:41.778788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2qzhg"
Sep 17 16:48:42.009038 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:42.009016    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz"]
Sep 17 16:48:42.010644 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:48:42.010616    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6211f1f7_b0dc_4d47_bef4_f51028d720ad.slice/crio-ef02602b1f8c3f5aa07e2fcd180f0fffb4553983e7b44d67087479cd3f525ceb WatchSource:0}: Error finding container ef02602b1f8c3f5aa07e2fcd180f0fffb4553983e7b44d67087479cd3f525ceb: Status 404 returned error can't find the container with id ef02602b1f8c3f5aa07e2fcd180f0fffb4553983e7b44d67087479cd3f525ceb
Sep 17 16:48:42.058401 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:42.058377    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2qzhg"]
Sep 17 16:48:42.061331 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:48:42.061307    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45a52e9_4d7d_464e_9ae6_9e73113e5585.slice/crio-2c7ca18a44d10f93fd3147dc7b4b25d6417e4ecf6151b20df78cb6501abdea48 WatchSource:0}: Error finding container 2c7ca18a44d10f93fd3147dc7b4b25d6417e4ecf6151b20df78cb6501abdea48: Status 404 returned error can't find the container with id 2c7ca18a44d10f93fd3147dc7b4b25d6417e4ecf6151b20df78cb6501abdea48
Sep 17 16:48:42.277349 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:42.277276    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2qzhg" event={"ID":"c45a52e9-4d7d-464e-9ae6-9e73113e5585","Type":"ContainerStarted","Data":"2c7ca18a44d10f93fd3147dc7b4b25d6417e4ecf6151b20df78cb6501abdea48"}
Sep 17 16:48:42.278472 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:42.278449    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz" event={"ID":"6211f1f7-b0dc-4d47-bef4-f51028d720ad","Type":"ContainerStarted","Data":"ef02602b1f8c3f5aa07e2fcd180f0fffb4553983e7b44d67087479cd3f525ceb"}
Sep 17 16:48:46.301129 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:46.301080    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2qzhg" event={"ID":"c45a52e9-4d7d-464e-9ae6-9e73113e5585","Type":"ContainerStarted","Data":"e0f5cb5990251916a05c785b82d8b7e5e3b8c957dee44d3d3acabe2124bc5717"}
Sep 17 16:48:46.301551 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:46.301150    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-2qzhg"
Sep 17 16:48:46.302622 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:46.302598    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz" event={"ID":"6211f1f7-b0dc-4d47-bef4-f51028d720ad","Type":"ContainerStarted","Data":"61c0b12b30cf57b5d5fcf421c21616e9f1ce2134c150507b65ec7592cfa5a335"}
Sep 17 16:48:46.302727 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:46.302719    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz"
Sep 17 16:48:46.317873 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:46.317833    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-2qzhg" podStartSLOduration=1.7379165250000002 podStartE2EDuration="5.317819715s" totalImagesPullingTime="3.57990319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:48:46.316826061 +0000 UTC m=+444.341734349" watchObservedRunningTime="2026-09-17 16:48:46.317819715 +0000 UTC m=+444.342728002"
Sep 17 16:48:46.331759 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:46.331721    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz" podStartSLOduration=1.788355874 podStartE2EDuration="5.331707695s" totalImagesPullingTime="3.543351821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:48:46.330869118 +0000 UTC m=+444.355777408" watchObservedRunningTime="2026-09-17 16:48:46.331707695 +0000 UTC m=+444.356615993"
Sep 17 16:48:52.309135 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:48:52.309102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-2qzhg"
Sep 17 16:49:17.309101 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:49:17.309071    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-7mlwz"
Sep 17 16:51:22.414391 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:51:22.414365    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 16:51:22.417948 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:51:22.417918    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 16:52:24.052496 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.052453    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"]
Sep 17 16:52:24.055775 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.055759    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.058309 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.058283    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:52:24.058525 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.058509    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-5ce245\""
Sep 17 16:52:24.058602 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.058558    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-5ce245-predictor-serving-cert\""
Sep 17 16:52:24.058602 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.058570    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-5ce245-dockercfg-52z8w\""
Sep 17 16:52:24.058602 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.058588    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:52:24.059462 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.059442    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-5ce245-kube-rbac-proxy-sar-config\""
Sep 17 16:52:24.059554 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.059486    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:52:24.063480 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:52:24.063438    2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613856fd_0018_4cfb_b0b2_d76fb768be3e.slice\": RecentStats: unable to find data in memory cache]"
Sep 17 16:52:24.065669 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.065648    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"]
Sep 17 16:52:24.163888 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.163863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/613856fd-0018-4cfb-b0b2-d76fb768be3e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-5ce245-predictor-54f49b45dd-wh622\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.163984 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.163897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljkb\" (UniqueName: \"kubernetes.io/projected/613856fd-0018-4cfb-b0b2-d76fb768be3e-kube-api-access-sljkb\") pod \"isvc-init-fail-5ce245-predictor-54f49b45dd-wh622\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.163984 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.163916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/613856fd-0018-4cfb-b0b2-d76fb768be3e-kserve-provision-location\") pod \"isvc-init-fail-5ce245-predictor-54f49b45dd-wh622\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.164059 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.163997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/613856fd-0018-4cfb-b0b2-d76fb768be3e-proxy-tls\") pod \"isvc-init-fail-5ce245-predictor-54f49b45dd-wh622\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.164059 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.164027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/613856fd-0018-4cfb-b0b2-d76fb768be3e-cabundle-cert\") pod \"isvc-init-fail-5ce245-predictor-54f49b45dd-wh622\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.265061 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.265038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/613856fd-0018-4cfb-b0b2-d76fb768be3e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-5ce245-predictor-54f49b45dd-wh622\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.265151 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.265070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljkb\" (UniqueName: \"kubernetes.io/projected/613856fd-0018-4cfb-b0b2-d76fb768be3e-kube-api-access-sljkb\") pod \"isvc-init-fail-5ce245-predictor-54f49b45dd-wh622\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.265151 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.265090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/613856fd-0018-4cfb-b0b2-d76fb768be3e-kserve-provision-location\") pod \"isvc-init-fail-5ce245-predictor-54f49b45dd-wh622\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.265151 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.265119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/613856fd-0018-4cfb-b0b2-d76fb768be3e-proxy-tls\") pod \"isvc-init-fail-5ce245-predictor-54f49b45dd-wh622\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.265151 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.265150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/613856fd-0018-4cfb-b0b2-d76fb768be3e-cabundle-cert\") pod \"isvc-init-fail-5ce245-predictor-54f49b45dd-wh622\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.265526 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.265509    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/613856fd-0018-4cfb-b0b2-d76fb768be3e-kserve-provision-location\") pod \"isvc-init-fail-5ce245-predictor-54f49b45dd-wh622\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.265728 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.265708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/613856fd-0018-4cfb-b0b2-d76fb768be3e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-5ce245-predictor-54f49b45dd-wh622\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.265821 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.265804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/613856fd-0018-4cfb-b0b2-d76fb768be3e-cabundle-cert\") pod \"isvc-init-fail-5ce245-predictor-54f49b45dd-wh622\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.267540 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.267525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/613856fd-0018-4cfb-b0b2-d76fb768be3e-proxy-tls\") pod \"isvc-init-fail-5ce245-predictor-54f49b45dd-wh622\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.273334 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.273310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljkb\" (UniqueName: \"kubernetes.io/projected/613856fd-0018-4cfb-b0b2-d76fb768be3e-kube-api-access-sljkb\") pod \"isvc-init-fail-5ce245-predictor-54f49b45dd-wh622\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.366315 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.366262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:24.637977 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.637944    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"]
Sep 17 16:52:24.640526 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:52:24.640501    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613856fd_0018_4cfb_b0b2_d76fb768be3e.slice/crio-2ac59ee5abe8c71ed1a7f891c431296f3f2af83e70493aec773a77807d7f0a9e WatchSource:0}: Error finding container 2ac59ee5abe8c71ed1a7f891c431296f3f2af83e70493aec773a77807d7f0a9e: Status 404 returned error can't find the container with id 2ac59ee5abe8c71ed1a7f891c431296f3f2af83e70493aec773a77807d7f0a9e
Sep 17 16:52:24.642890 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:24.642872    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:52:25.227988 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:25.227953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622" event={"ID":"613856fd-0018-4cfb-b0b2-d76fb768be3e","Type":"ContainerStarted","Data":"2ac59ee5abe8c71ed1a7f891c431296f3f2af83e70493aec773a77807d7f0a9e"}
Sep 17 16:52:28.243399 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:28.243364    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622" event={"ID":"613856fd-0018-4cfb-b0b2-d76fb768be3e","Type":"ContainerStarted","Data":"f532868da7212dc37afbf2b582ee0b34bb4a288b7dce059e57ab95363c6d4a4f"}
Sep 17 16:52:34.271079 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:34.271049    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5ce245-predictor-54f49b45dd-wh622_613856fd-0018-4cfb-b0b2-d76fb768be3e/storage-initializer/0.log"
Sep 17 16:52:34.271502 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:34.271089    2577 generic.go:356] "Generic (PLEG): container finished" podID="613856fd-0018-4cfb-b0b2-d76fb768be3e" containerID="f532868da7212dc37afbf2b582ee0b34bb4a288b7dce059e57ab95363c6d4a4f" exitCode=1
Sep 17 16:52:34.271502 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:34.271186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622" event={"ID":"613856fd-0018-4cfb-b0b2-d76fb768be3e","Type":"ContainerDied","Data":"f532868da7212dc37afbf2b582ee0b34bb4a288b7dce059e57ab95363c6d4a4f"}
Sep 17 16:52:35.277378 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:35.277346    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5ce245-predictor-54f49b45dd-wh622_613856fd-0018-4cfb-b0b2-d76fb768be3e/storage-initializer/0.log"
Sep 17 16:52:35.277758 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:35.277475    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622" event={"ID":"613856fd-0018-4cfb-b0b2-d76fb768be3e","Type":"ContainerStarted","Data":"7883584111b161b56ffa5e1cb6fbd3d74cc6b4584759881d23f379700c24a499"}
Sep 17 16:52:38.758669 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:38.758635    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"]
Sep 17 16:52:38.759090 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:38.758850    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622" podUID="613856fd-0018-4cfb-b0b2-d76fb768be3e" containerName="storage-initializer" containerID="cri-o://7883584111b161b56ffa5e1cb6fbd3d74cc6b4584759881d23f379700c24a499" gracePeriod=30
Sep 17 16:52:39.901058 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:39.901035    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5ce245-predictor-54f49b45dd-wh622_613856fd-0018-4cfb-b0b2-d76fb768be3e/storage-initializer/1.log"
Sep 17 16:52:39.901419 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:39.901405    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5ce245-predictor-54f49b45dd-wh622_613856fd-0018-4cfb-b0b2-d76fb768be3e/storage-initializer/0.log"
Sep 17 16:52:39.903734 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:39.903718    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:39.978120 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:39.978062    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sljkb\" (UniqueName: \"kubernetes.io/projected/613856fd-0018-4cfb-b0b2-d76fb768be3e-kube-api-access-sljkb\") pod \"613856fd-0018-4cfb-b0b2-d76fb768be3e\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") "
Sep 17 16:52:39.978120 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:39.978112    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/613856fd-0018-4cfb-b0b2-d76fb768be3e-cabundle-cert\") pod \"613856fd-0018-4cfb-b0b2-d76fb768be3e\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") "
Sep 17 16:52:39.978317 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:39.978135    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/613856fd-0018-4cfb-b0b2-d76fb768be3e-kube-rbac-proxy-sar-config\") pod \"613856fd-0018-4cfb-b0b2-d76fb768be3e\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") "
Sep 17 16:52:39.978317 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:39.978182    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/613856fd-0018-4cfb-b0b2-d76fb768be3e-kserve-provision-location\") pod \"613856fd-0018-4cfb-b0b2-d76fb768be3e\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") "
Sep 17 16:52:39.978317 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:39.978217    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/613856fd-0018-4cfb-b0b2-d76fb768be3e-proxy-tls\") pod \"613856fd-0018-4cfb-b0b2-d76fb768be3e\" (UID: \"613856fd-0018-4cfb-b0b2-d76fb768be3e\") "
Sep 17 16:52:39.978503 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:39.978479    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613856fd-0018-4cfb-b0b2-d76fb768be3e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "613856fd-0018-4cfb-b0b2-d76fb768be3e" (UID: "613856fd-0018-4cfb-b0b2-d76fb768be3e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:52:39.978563 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:39.978511    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613856fd-0018-4cfb-b0b2-d76fb768be3e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "613856fd-0018-4cfb-b0b2-d76fb768be3e" (UID: "613856fd-0018-4cfb-b0b2-d76fb768be3e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:52:39.978614 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:39.978556    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613856fd-0018-4cfb-b0b2-d76fb768be3e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "613856fd-0018-4cfb-b0b2-d76fb768be3e" (UID: "613856fd-0018-4cfb-b0b2-d76fb768be3e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:52:39.980170 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:39.980132    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/613856fd-0018-4cfb-b0b2-d76fb768be3e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "613856fd-0018-4cfb-b0b2-d76fb768be3e" (UID: "613856fd-0018-4cfb-b0b2-d76fb768be3e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:52:39.980334 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:39.980318    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613856fd-0018-4cfb-b0b2-d76fb768be3e-kube-api-access-sljkb" (OuterVolumeSpecName: "kube-api-access-sljkb") pod "613856fd-0018-4cfb-b0b2-d76fb768be3e" (UID: "613856fd-0018-4cfb-b0b2-d76fb768be3e"). InnerVolumeSpecName "kube-api-access-sljkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:52:40.079352 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.079324    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljkb\" (UniqueName: \"kubernetes.io/projected/613856fd-0018-4cfb-b0b2-d76fb768be3e-kube-api-access-sljkb\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:52:40.079352 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.079346    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/613856fd-0018-4cfb-b0b2-d76fb768be3e-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:52:40.079352 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.079355    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/613856fd-0018-4cfb-b0b2-d76fb768be3e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:52:40.079533 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.079364    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/613856fd-0018-4cfb-b0b2-d76fb768be3e-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:52:40.079533 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.079393    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/613856fd-0018-4cfb-b0b2-d76fb768be3e-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:52:40.303540 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.303515    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5ce245-predictor-54f49b45dd-wh622_613856fd-0018-4cfb-b0b2-d76fb768be3e/storage-initializer/1.log"
Sep 17 16:52:40.303967 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.303947    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5ce245-predictor-54f49b45dd-wh622_613856fd-0018-4cfb-b0b2-d76fb768be3e/storage-initializer/0.log"
Sep 17 16:52:40.304063 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.303992    2577 generic.go:356] "Generic (PLEG): container finished" podID="613856fd-0018-4cfb-b0b2-d76fb768be3e" containerID="7883584111b161b56ffa5e1cb6fbd3d74cc6b4584759881d23f379700c24a499" exitCode=1
Sep 17 16:52:40.304063 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.304035    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622" event={"ID":"613856fd-0018-4cfb-b0b2-d76fb768be3e","Type":"ContainerDied","Data":"7883584111b161b56ffa5e1cb6fbd3d74cc6b4584759881d23f379700c24a499"}
Sep 17 16:52:40.304063 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.304050    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"
Sep 17 16:52:40.304236 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.304140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622" event={"ID":"613856fd-0018-4cfb-b0b2-d76fb768be3e","Type":"ContainerDied","Data":"2ac59ee5abe8c71ed1a7f891c431296f3f2af83e70493aec773a77807d7f0a9e"}
Sep 17 16:52:40.304236 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.304211    2577 scope.go:117] "RemoveContainer" containerID="7883584111b161b56ffa5e1cb6fbd3d74cc6b4584759881d23f379700c24a499"
Sep 17 16:52:40.313129 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.313108    2577 scope.go:117] "RemoveContainer" containerID="f532868da7212dc37afbf2b582ee0b34bb4a288b7dce059e57ab95363c6d4a4f"
Sep 17 16:52:40.320935 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.320918    2577 scope.go:117] "RemoveContainer" containerID="7883584111b161b56ffa5e1cb6fbd3d74cc6b4584759881d23f379700c24a499"
Sep 17 16:52:40.321222 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:52:40.321200    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7883584111b161b56ffa5e1cb6fbd3d74cc6b4584759881d23f379700c24a499\": container with ID starting with 7883584111b161b56ffa5e1cb6fbd3d74cc6b4584759881d23f379700c24a499 not found: ID does not exist" containerID="7883584111b161b56ffa5e1cb6fbd3d74cc6b4584759881d23f379700c24a499"
Sep 17 16:52:40.321266 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.321231    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7883584111b161b56ffa5e1cb6fbd3d74cc6b4584759881d23f379700c24a499"} err="failed to get container status \"7883584111b161b56ffa5e1cb6fbd3d74cc6b4584759881d23f379700c24a499\": rpc error: code = NotFound desc = could not find container \"7883584111b161b56ffa5e1cb6fbd3d74cc6b4584759881d23f379700c24a499\": container with ID starting with 7883584111b161b56ffa5e1cb6fbd3d74cc6b4584759881d23f379700c24a499 not found: ID does not exist"
Sep 17 16:52:40.321266 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.321247    2577 scope.go:117] "RemoveContainer" containerID="f532868da7212dc37afbf2b582ee0b34bb4a288b7dce059e57ab95363c6d4a4f"
Sep 17 16:52:40.321548 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:52:40.321530    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f532868da7212dc37afbf2b582ee0b34bb4a288b7dce059e57ab95363c6d4a4f\": container with ID starting with f532868da7212dc37afbf2b582ee0b34bb4a288b7dce059e57ab95363c6d4a4f not found: ID does not exist" containerID="f532868da7212dc37afbf2b582ee0b34bb4a288b7dce059e57ab95363c6d4a4f"
Sep 17 16:52:40.321608 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.321555    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f532868da7212dc37afbf2b582ee0b34bb4a288b7dce059e57ab95363c6d4a4f"} err="failed to get container status \"f532868da7212dc37afbf2b582ee0b34bb4a288b7dce059e57ab95363c6d4a4f\": rpc error: code = NotFound desc = could not find container \"f532868da7212dc37afbf2b582ee0b34bb4a288b7dce059e57ab95363c6d4a4f\": container with ID starting with f532868da7212dc37afbf2b582ee0b34bb4a288b7dce059e57ab95363c6d4a4f not found: ID does not exist"
Sep 17 16:52:40.342061 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.342034    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"]
Sep 17 16:52:40.348072 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.348044    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5ce245-predictor-54f49b45dd-wh622"]
Sep 17 16:52:40.446827 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:52:40.446803    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="613856fd-0018-4cfb-b0b2-d76fb768be3e" path="/var/lib/kubelet/pods/613856fd-0018-4cfb-b0b2-d76fb768be3e/volumes"
Sep 17 16:54:39.722211 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.722179    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"]
Sep 17 16:54:39.722697 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.722520    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="613856fd-0018-4cfb-b0b2-d76fb768be3e" containerName="storage-initializer"
Sep 17 16:54:39.722697 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.722533    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="613856fd-0018-4cfb-b0b2-d76fb768be3e" containerName="storage-initializer"
Sep 17 16:54:39.722697 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.722541    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="613856fd-0018-4cfb-b0b2-d76fb768be3e" containerName="storage-initializer"
Sep 17 16:54:39.722697 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.722547    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="613856fd-0018-4cfb-b0b2-d76fb768be3e" containerName="storage-initializer"
Sep 17 16:54:39.722697 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.722604    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="613856fd-0018-4cfb-b0b2-d76fb768be3e" containerName="storage-initializer"
Sep 17 16:54:39.722872 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.722702    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="613856fd-0018-4cfb-b0b2-d76fb768be3e" containerName="storage-initializer"
Sep 17 16:54:39.725853 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.725838    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:39.728410 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.728387    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 17 16:54:39.728410 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.728406    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:54:39.728585 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.728534    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 16:54:39.728771 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.728754    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:54:39.729453 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.729433    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 16:54:39.729762 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.729737    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:54:39.729850 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.729765    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:54:39.734409 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.734382    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"]
Sep 17 16:54:39.802569 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.802539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:39.802695 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.802599    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wtlv\" (UniqueName: \"kubernetes.io/projected/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kube-api-access-4wtlv\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:39.802695 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.802651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/514f5dda-a1b1-4f9a-9d82-7705f119d37b-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:39.802695 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.802672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/514f5dda-a1b1-4f9a-9d82-7705f119d37b-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:39.802795 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.802714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:39.903655 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.903624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wtlv\" (UniqueName: \"kubernetes.io/projected/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kube-api-access-4wtlv\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:39.903817 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.903672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/514f5dda-a1b1-4f9a-9d82-7705f119d37b-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:39.903817 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.903692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/514f5dda-a1b1-4f9a-9d82-7705f119d37b-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:39.903817 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.903709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:39.903978 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:54:39.903860    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 17 16:54:39.903978 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:54:39.903946    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514f5dda-a1b1-4f9a-9d82-7705f119d37b-proxy-tls podName:514f5dda-a1b1-4f9a-9d82-7705f119d37b nodeName:}" failed. No retries permitted until 2026-09-17 16:54:40.403923691 +0000 UTC m=+798.428831960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/514f5dda-a1b1-4f9a-9d82-7705f119d37b-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-78mk4" (UID: "514f5dda-a1b1-4f9a-9d82-7705f119d37b") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 17 16:54:39.904079 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.903971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:39.904079 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.904008    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:39.904349 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.904330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/514f5dda-a1b1-4f9a-9d82-7705f119d37b-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:39.904577 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.904560    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:39.912305 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:39.912288    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wtlv\" (UniqueName: \"kubernetes.io/projected/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kube-api-access-4wtlv\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:40.408576 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:40.408531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/514f5dda-a1b1-4f9a-9d82-7705f119d37b-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:40.410872 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:40.410844    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/514f5dda-a1b1-4f9a-9d82-7705f119d37b-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-78mk4\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:40.637377 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:40.637337    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:54:40.912792 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:40.912759    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"]
Sep 17 16:54:40.914376 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:54:40.914346    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514f5dda_a1b1_4f9a_9d82_7705f119d37b.slice/crio-de6f031a5a4273131b1e723fbed2ac250e21b6eb836deb26c40c5f69c0e3e309 WatchSource:0}: Error finding container de6f031a5a4273131b1e723fbed2ac250e21b6eb836deb26c40c5f69c0e3e309: Status 404 returned error can't find the container with id de6f031a5a4273131b1e723fbed2ac250e21b6eb836deb26c40c5f69c0e3e309
Sep 17 16:54:41.824527 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:41.824487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" event={"ID":"514f5dda-a1b1-4f9a-9d82-7705f119d37b","Type":"ContainerStarted","Data":"1f885fa79bfc789cb58e0f8bb07214a887161a082b46c1e5dcde8e5adafa3c1b"}
Sep 17 16:54:41.824697 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:41.824560    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" event={"ID":"514f5dda-a1b1-4f9a-9d82-7705f119d37b","Type":"ContainerStarted","Data":"de6f031a5a4273131b1e723fbed2ac250e21b6eb836deb26c40c5f69c0e3e309"}
Sep 17 16:54:44.838083 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:44.838051    2577 generic.go:356] "Generic (PLEG): container finished" podID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerID="1f885fa79bfc789cb58e0f8bb07214a887161a082b46c1e5dcde8e5adafa3c1b" exitCode=0
Sep 17 16:54:44.838392 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:54:44.838126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" event={"ID":"514f5dda-a1b1-4f9a-9d82-7705f119d37b","Type":"ContainerDied","Data":"1f885fa79bfc789cb58e0f8bb07214a887161a082b46c1e5dcde8e5adafa3c1b"}
Sep 17 16:55:06.965269 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:55:06.965207    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" event={"ID":"514f5dda-a1b1-4f9a-9d82-7705f119d37b","Type":"ContainerStarted","Data":"b0d3c14d1d521694b30e2dc9d00bf498798f6b0d57124f29edd224f86bf54338"}
Sep 17 16:55:09.981468 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:55:09.981434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" event={"ID":"514f5dda-a1b1-4f9a-9d82-7705f119d37b","Type":"ContainerStarted","Data":"5fa77ace9bb46db45a14ad03e29987d01f17585f22c2c4f129702e860cf15f2d"}
Sep 17 16:55:09.981800 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:55:09.981574    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:55:10.000059 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:55:10.000011    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" podStartSLOduration=2.405086293 podStartE2EDuration="30.999995624s" totalImagesPullingTime="24.781084697s" totalInitContainerRuntime="3.813824634s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:54:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:55:09.997820375 +0000 UTC m=+828.022728678" watchObservedRunningTime="2026-09-17 16:55:09.999995624 +0000 UTC m=+828.024903912"
Sep 17 16:55:10.986049 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:55:10.986021    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:55:10.987426 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:55:10.987375    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:55:11.990142 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:55:11.990099    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:55:16.995809 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:55:16.995775    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:55:16.996386 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:55:16.996355    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:55:26.996502 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:55:26.996457    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:55:36.997305 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:55:36.997263    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:55:46.996305 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:55:46.996263    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:55:56.997217 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:55:56.997179    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:56:06.997137 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:06.997092    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:56:16.997265 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:16.997184    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:56:22.444665 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:22.444624    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 16:56:22.445102 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:22.444621    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 16:56:26.996932 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:26.996899    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:56:30.219280 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:30.219249    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"]
Sep 17 16:56:30.219781 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:30.219443    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kserve-container" containerID="cri-o://b0d3c14d1d521694b30e2dc9d00bf498798f6b0d57124f29edd224f86bf54338" gracePeriod=30
Sep 17 16:56:30.219781 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:30.219503    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kube-rbac-proxy" containerID="cri-o://5fa77ace9bb46db45a14ad03e29987d01f17585f22c2c4f129702e860cf15f2d" gracePeriod=30
Sep 17 16:56:31.332384 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:31.332349    2577 generic.go:356] "Generic (PLEG): container finished" podID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerID="5fa77ace9bb46db45a14ad03e29987d01f17585f22c2c4f129702e860cf15f2d" exitCode=2
Sep 17 16:56:31.332732 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:31.332423    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" event={"ID":"514f5dda-a1b1-4f9a-9d82-7705f119d37b","Type":"ContainerDied","Data":"5fa77ace9bb46db45a14ad03e29987d01f17585f22c2c4f129702e860cf15f2d"}
Sep 17 16:56:31.990474 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:31.990426    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 16:56:34.348060 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.348034    2577 generic.go:356] "Generic (PLEG): container finished" podID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerID="b0d3c14d1d521694b30e2dc9d00bf498798f6b0d57124f29edd224f86bf54338" exitCode=0
Sep 17 16:56:34.348386 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.348095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" event={"ID":"514f5dda-a1b1-4f9a-9d82-7705f119d37b","Type":"ContainerDied","Data":"b0d3c14d1d521694b30e2dc9d00bf498798f6b0d57124f29edd224f86bf54338"}
Sep 17 16:56:34.364436 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.364418    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:56:34.474563 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.474506    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wtlv\" (UniqueName: \"kubernetes.io/projected/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kube-api-access-4wtlv\") pod \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") "
Sep 17 16:56:34.474563 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.474544    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/514f5dda-a1b1-4f9a-9d82-7705f119d37b-proxy-tls\") pod \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") "
Sep 17 16:56:34.474700 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.474634    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/514f5dda-a1b1-4f9a-9d82-7705f119d37b-cabundle-cert\") pod \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") "
Sep 17 16:56:34.474700 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.474670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kube-rbac-proxy-sar-config\") pod \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") "
Sep 17 16:56:34.474700 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.474695    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kserve-provision-location\") pod \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\" (UID: \"514f5dda-a1b1-4f9a-9d82-7705f119d37b\") "
Sep 17 16:56:34.475061 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.475029    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "514f5dda-a1b1-4f9a-9d82-7705f119d37b" (UID: "514f5dda-a1b1-4f9a-9d82-7705f119d37b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:56:34.475149 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.475038    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514f5dda-a1b1-4f9a-9d82-7705f119d37b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "514f5dda-a1b1-4f9a-9d82-7705f119d37b" (UID: "514f5dda-a1b1-4f9a-9d82-7705f119d37b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:56:34.475149 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.475060    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "514f5dda-a1b1-4f9a-9d82-7705f119d37b" (UID: "514f5dda-a1b1-4f9a-9d82-7705f119d37b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:56:34.477082 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.477063    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514f5dda-a1b1-4f9a-9d82-7705f119d37b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "514f5dda-a1b1-4f9a-9d82-7705f119d37b" (UID: "514f5dda-a1b1-4f9a-9d82-7705f119d37b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:56:34.477168 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.477077    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kube-api-access-4wtlv" (OuterVolumeSpecName: "kube-api-access-4wtlv") pod "514f5dda-a1b1-4f9a-9d82-7705f119d37b" (UID: "514f5dda-a1b1-4f9a-9d82-7705f119d37b"). InnerVolumeSpecName "kube-api-access-4wtlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:56:34.576084 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.576060    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wtlv\" (UniqueName: \"kubernetes.io/projected/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kube-api-access-4wtlv\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:56:34.576084 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.576081    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/514f5dda-a1b1-4f9a-9d82-7705f119d37b-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:56:34.576229 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.576090    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/514f5dda-a1b1-4f9a-9d82-7705f119d37b-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:56:34.576229 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.576098    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:56:34.576229 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:34.576106    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/514f5dda-a1b1-4f9a-9d82-7705f119d37b-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:56:35.355338 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.355311    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"
Sep 17 16:56:35.355804 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.355311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4" event={"ID":"514f5dda-a1b1-4f9a-9d82-7705f119d37b","Type":"ContainerDied","Data":"de6f031a5a4273131b1e723fbed2ac250e21b6eb836deb26c40c5f69c0e3e309"}
Sep 17 16:56:35.355804 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.355454    2577 scope.go:117] "RemoveContainer" containerID="5fa77ace9bb46db45a14ad03e29987d01f17585f22c2c4f129702e860cf15f2d"
Sep 17 16:56:35.364656 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.364639    2577 scope.go:117] "RemoveContainer" containerID="b0d3c14d1d521694b30e2dc9d00bf498798f6b0d57124f29edd224f86bf54338"
Sep 17 16:56:35.373716 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.373695    2577 scope.go:117] "RemoveContainer" containerID="1f885fa79bfc789cb58e0f8bb07214a887161a082b46c1e5dcde8e5adafa3c1b"
Sep 17 16:56:35.376827 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.376807    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"]
Sep 17 16:56:35.380356 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.380335    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-78mk4"]
Sep 17 16:56:35.393268 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.393250    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"]
Sep 17 16:56:35.393556 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.393545    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kserve-container"
Sep 17 16:56:35.393600 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.393558    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kserve-container"
Sep 17 16:56:35.393600 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.393577    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kube-rbac-proxy"
Sep 17 16:56:35.393600 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.393583    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kube-rbac-proxy"
Sep 17 16:56:35.393600 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.393592    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="storage-initializer"
Sep 17 16:56:35.393600 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.393598    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="storage-initializer"
Sep 17 16:56:35.393744 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.393642    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kserve-container"
Sep 17 16:56:35.393744 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.393651    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" containerName="kube-rbac-proxy"
Sep 17 16:56:35.397680 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.397666    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:35.400233 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.400210    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:56:35.400311 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.400236    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 16:56:35.400311 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.400219    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 17 16:56:35.400524 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.400509    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:56:35.400589 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.400543    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 16:56:35.400662 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.400645    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:56:35.400720 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.400704    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:56:35.406486 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.406467    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"]
Sep 17 16:56:35.486921 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.482585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:35.486921 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.482649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feebd8bc-0adc-43a9-accd-ba3024bd19ea-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:35.486921 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.482720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feebd8bc-0adc-43a9-accd-ba3024bd19ea-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:35.486921 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.482758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:35.486921 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.482805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slttq\" (UniqueName: \"kubernetes.io/projected/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kube-api-access-slttq\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:35.583666 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.583644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:35.583762 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.583681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slttq\" (UniqueName: \"kubernetes.io/projected/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kube-api-access-slttq\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:35.583762 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.583701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:35.583762 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.583724    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feebd8bc-0adc-43a9-accd-ba3024bd19ea-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:35.583879 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.583778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feebd8bc-0adc-43a9-accd-ba3024bd19ea-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:35.583879 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:56:35.583875    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 17 16:56:35.583963 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:56:35.583931    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feebd8bc-0adc-43a9-accd-ba3024bd19ea-proxy-tls podName:feebd8bc-0adc-43a9-accd-ba3024bd19ea nodeName:}" failed. No retries permitted until 2026-09-17 16:56:36.083912297 +0000 UTC m=+914.108820578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/feebd8bc-0adc-43a9-accd-ba3024bd19ea-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" (UID: "feebd8bc-0adc-43a9-accd-ba3024bd19ea") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 17 16:56:35.584093 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.584074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:35.584425 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.584408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:35.584466 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.584410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feebd8bc-0adc-43a9-accd-ba3024bd19ea-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:35.594369 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:35.594349    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slttq\" (UniqueName: \"kubernetes.io/projected/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kube-api-access-slttq\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:36.086920 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:36.086883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feebd8bc-0adc-43a9-accd-ba3024bd19ea-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:36.089372 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:36.089349    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feebd8bc-0adc-43a9-accd-ba3024bd19ea-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:36.308741 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:36.308712    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:36.447472 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:36.447437    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="514f5dda-a1b1-4f9a-9d82-7705f119d37b" path="/var/lib/kubelet/pods/514f5dda-a1b1-4f9a-9d82-7705f119d37b/volumes"
Sep 17 16:56:36.581644 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:36.581620    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"]
Sep 17 16:56:36.583856 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:56:36.583827    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeebd8bc_0adc_43a9_accd_ba3024bd19ea.slice/crio-dcf8e84afc3cd2a38dc3dea472d053b61f6285fb33997035d44ba538873ff9f5 WatchSource:0}: Error finding container dcf8e84afc3cd2a38dc3dea472d053b61f6285fb33997035d44ba538873ff9f5: Status 404 returned error can't find the container with id dcf8e84afc3cd2a38dc3dea472d053b61f6285fb33997035d44ba538873ff9f5
Sep 17 16:56:37.371288 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:37.371250    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" event={"ID":"feebd8bc-0adc-43a9-accd-ba3024bd19ea","Type":"ContainerStarted","Data":"7f200848c658f82983561d69f5f67dcf4ca3783311dccf1c729cd7203d414cbf"}
Sep 17 16:56:37.371446 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:37.371313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" event={"ID":"feebd8bc-0adc-43a9-accd-ba3024bd19ea","Type":"ContainerStarted","Data":"dcf8e84afc3cd2a38dc3dea472d053b61f6285fb33997035d44ba538873ff9f5"}
Sep 17 16:56:40.386303 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:40.386271    2577 generic.go:356] "Generic (PLEG): container finished" podID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerID="7f200848c658f82983561d69f5f67dcf4ca3783311dccf1c729cd7203d414cbf" exitCode=0
Sep 17 16:56:40.386708 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:40.386341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" event={"ID":"feebd8bc-0adc-43a9-accd-ba3024bd19ea","Type":"ContainerDied","Data":"7f200848c658f82983561d69f5f67dcf4ca3783311dccf1c729cd7203d414cbf"}
Sep 17 16:56:41.392812 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:41.392780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" event={"ID":"feebd8bc-0adc-43a9-accd-ba3024bd19ea","Type":"ContainerStarted","Data":"3fb3ad7e7f459eb05cd9b068aa70df5e5f279761cf07cb15f5fe691f9e6f7e44"}
Sep 17 16:56:41.393283 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:41.392847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" event={"ID":"feebd8bc-0adc-43a9-accd-ba3024bd19ea","Type":"ContainerStarted","Data":"54fd0385d77f364c22ccb194135ac038eed52141a833fa5496730993ebd813c5"}
Sep 17 16:56:41.393283 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:41.393053    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:41.413408 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:41.413364    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" podStartSLOduration=2.782039105 podStartE2EDuration="6.413351951s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.631312846s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:56:41.41128704 +0000 UTC m=+919.436195327" watchObservedRunningTime="2026-09-17 16:56:41.413351951 +0000 UTC m=+919.438260238"
Sep 17 16:56:42.397653 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:42.397616    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:42.398771 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:42.398746    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:56:43.402298 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:43.402260    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:56:48.406645 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:48.406614    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:56:48.407294 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:48.407258    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:56:58.407324 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:56:58.407272    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:57:08.407322 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:08.407277    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:57:18.407993 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:18.407940    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:57:28.407467 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:28.407429    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:57:38.407771 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:38.407726    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:57:39.437887 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.437846    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"]
Sep 17 16:57:39.442815 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.442785    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:39.445235 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.445206    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 17 16:57:39.445315 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.445264    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 17 16:57:39.450959 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.450936    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"]
Sep 17 16:57:39.537318 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.537291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403e2840-08bb-40f5-a5d9-d9089cbaefb1-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:39.537414 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.537323    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:39.537414 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.537339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhnp5\" (UniqueName: \"kubernetes.io/projected/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kube-api-access-mhnp5\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:39.537414 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.537354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/403e2840-08bb-40f5-a5d9-d9089cbaefb1-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:39.537558 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.537440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:39.638758 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.638734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403e2840-08bb-40f5-a5d9-d9089cbaefb1-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:39.638874 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.638764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:39.638874 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.638780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhnp5\" (UniqueName: \"kubernetes.io/projected/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kube-api-access-mhnp5\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:39.638874 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.638797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/403e2840-08bb-40f5-a5d9-d9089cbaefb1-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:39.638874 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.638835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:39.639094 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:57:39.638914    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Sep 17 16:57:39.639094 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:57:39.638985    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403e2840-08bb-40f5-a5d9-d9089cbaefb1-proxy-tls podName:403e2840-08bb-40f5-a5d9-d9089cbaefb1 nodeName:}" failed. No retries permitted until 2026-09-17 16:57:40.138970031 +0000 UTC m=+978.163878297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/403e2840-08bb-40f5-a5d9-d9089cbaefb1-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-sjzdz" (UID: "403e2840-08bb-40f5-a5d9-d9089cbaefb1") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Sep 17 16:57:39.639300 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.639279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:39.639537 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.639518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:39.639599 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.639574    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/403e2840-08bb-40f5-a5d9-d9089cbaefb1-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:39.646647 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:39.646626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhnp5\" (UniqueName: \"kubernetes.io/projected/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kube-api-access-mhnp5\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:40.143056 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:40.143012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403e2840-08bb-40f5-a5d9-d9089cbaefb1-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:40.145537 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:40.145511    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403e2840-08bb-40f5-a5d9-d9089cbaefb1-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-sjzdz\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:40.353309 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:40.353273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:57:40.625264 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:40.625237    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"]
Sep 17 16:57:40.627769 ip-10-0-135-124 kubenswrapper[2577]: W0917 16:57:40.627743    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403e2840_08bb_40f5_a5d9_d9089cbaefb1.slice/crio-6d9661df17b7aa41854457e06eaba3570e77cb4e1ba52165168cee2cb8e30d99 WatchSource:0}: Error finding container 6d9661df17b7aa41854457e06eaba3570e77cb4e1ba52165168cee2cb8e30d99: Status 404 returned error can't find the container with id 6d9661df17b7aa41854457e06eaba3570e77cb4e1ba52165168cee2cb8e30d99
Sep 17 16:57:40.649281 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:40.649258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz" event={"ID":"403e2840-08bb-40f5-a5d9-d9089cbaefb1","Type":"ContainerStarted","Data":"6d9661df17b7aa41854457e06eaba3570e77cb4e1ba52165168cee2cb8e30d99"}
Sep 17 16:57:41.655216 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:41.655180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz" event={"ID":"403e2840-08bb-40f5-a5d9-d9089cbaefb1","Type":"ContainerStarted","Data":"3cd8b98f52cf775fecb4c04fe99b7e6a1fb1a8002a7fba40a37dfec0d0ee2e05"}
Sep 17 16:57:45.673987 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:45.673959    2577 generic.go:356] "Generic (PLEG): container finished" podID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerID="3cd8b98f52cf775fecb4c04fe99b7e6a1fb1a8002a7fba40a37dfec0d0ee2e05" exitCode=0
Sep 17 16:57:45.674297 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:45.674029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz" event={"ID":"403e2840-08bb-40f5-a5d9-d9089cbaefb1","Type":"ContainerDied","Data":"3cd8b98f52cf775fecb4c04fe99b7e6a1fb1a8002a7fba40a37dfec0d0ee2e05"}
Sep 17 16:57:45.675214 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:45.675198    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:57:48.407387 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:48.407338    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:57:50.449287 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:57:50.449244    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:58:00.449976 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:00.449949    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:58:06.055908 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:06.055867    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"]
Sep 17 16:58:06.056453 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:06.056114    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kserve-container" containerID="cri-o://54fd0385d77f364c22ccb194135ac038eed52141a833fa5496730993ebd813c5" gracePeriod=30
Sep 17 16:58:06.056453 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:06.056286    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kube-rbac-proxy" containerID="cri-o://3fb3ad7e7f459eb05cd9b068aa70df5e5f279761cf07cb15f5fe691f9e6f7e44" gracePeriod=30
Sep 17 16:58:06.785359 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:06.785324    2577 generic.go:356] "Generic (PLEG): container finished" podID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerID="3fb3ad7e7f459eb05cd9b068aa70df5e5f279761cf07cb15f5fe691f9e6f7e44" exitCode=2
Sep 17 16:58:06.785526 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:06.785394    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" event={"ID":"feebd8bc-0adc-43a9-accd-ba3024bd19ea","Type":"ContainerDied","Data":"3fb3ad7e7f459eb05cd9b068aa70df5e5f279761cf07cb15f5fe691f9e6f7e44"}
Sep 17 16:58:08.403605 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:08.403562    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 16:58:10.449086 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:10.449040    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:58:11.419999 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.419976    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:58:11.503443 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.503412    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feebd8bc-0adc-43a9-accd-ba3024bd19ea-cabundle-cert\") pod \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") "
Sep 17 16:58:11.503835 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.503463    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slttq\" (UniqueName: \"kubernetes.io/projected/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kube-api-access-slttq\") pod \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") "
Sep 17 16:58:11.503835 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.503484    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feebd8bc-0adc-43a9-accd-ba3024bd19ea-proxy-tls\") pod \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") "
Sep 17 16:58:11.503835 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.503528    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kube-rbac-proxy-sar-config\") pod \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") "
Sep 17 16:58:11.503835 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.503583    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kserve-provision-location\") pod \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\" (UID: \"feebd8bc-0adc-43a9-accd-ba3024bd19ea\") "
Sep 17 16:58:11.504027 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.503852    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feebd8bc-0adc-43a9-accd-ba3024bd19ea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "feebd8bc-0adc-43a9-accd-ba3024bd19ea" (UID: "feebd8bc-0adc-43a9-accd-ba3024bd19ea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:58:11.504027 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.503926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "feebd8bc-0adc-43a9-accd-ba3024bd19ea" (UID: "feebd8bc-0adc-43a9-accd-ba3024bd19ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:58:11.504027 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.503929    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "feebd8bc-0adc-43a9-accd-ba3024bd19ea" (UID: "feebd8bc-0adc-43a9-accd-ba3024bd19ea"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:58:11.505533 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.505516    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feebd8bc-0adc-43a9-accd-ba3024bd19ea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "feebd8bc-0adc-43a9-accd-ba3024bd19ea" (UID: "feebd8bc-0adc-43a9-accd-ba3024bd19ea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:58:11.505654 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.505634    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kube-api-access-slttq" (OuterVolumeSpecName: "kube-api-access-slttq") pod "feebd8bc-0adc-43a9-accd-ba3024bd19ea" (UID: "feebd8bc-0adc-43a9-accd-ba3024bd19ea"). InnerVolumeSpecName "kube-api-access-slttq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:58:11.604136 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.604109    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feebd8bc-0adc-43a9-accd-ba3024bd19ea-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:58:11.604279 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.604138    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slttq\" (UniqueName: \"kubernetes.io/projected/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kube-api-access-slttq\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:58:11.604279 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.604151    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feebd8bc-0adc-43a9-accd-ba3024bd19ea-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:58:11.604279 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.604179    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:58:11.604279 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.604191    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feebd8bc-0adc-43a9-accd-ba3024bd19ea-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 16:58:11.809954 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.809914    2577 generic.go:356] "Generic (PLEG): container finished" podID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerID="54fd0385d77f364c22ccb194135ac038eed52141a833fa5496730993ebd813c5" exitCode=0
Sep 17 16:58:11.810096 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.809963    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"
Sep 17 16:58:11.810096 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.809995    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" event={"ID":"feebd8bc-0adc-43a9-accd-ba3024bd19ea","Type":"ContainerDied","Data":"54fd0385d77f364c22ccb194135ac038eed52141a833fa5496730993ebd813c5"}
Sep 17 16:58:11.810096 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.810049    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs" event={"ID":"feebd8bc-0adc-43a9-accd-ba3024bd19ea","Type":"ContainerDied","Data":"dcf8e84afc3cd2a38dc3dea472d053b61f6285fb33997035d44ba538873ff9f5"}
Sep 17 16:58:11.810096 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.810076    2577 scope.go:117] "RemoveContainer" containerID="3fb3ad7e7f459eb05cd9b068aa70df5e5f279761cf07cb15f5fe691f9e6f7e44"
Sep 17 16:58:11.818579 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.818563    2577 scope.go:117] "RemoveContainer" containerID="54fd0385d77f364c22ccb194135ac038eed52141a833fa5496730993ebd813c5"
Sep 17 16:58:11.825661 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.825640    2577 scope.go:117] "RemoveContainer" containerID="7f200848c658f82983561d69f5f67dcf4ca3783311dccf1c729cd7203d414cbf"
Sep 17 16:58:11.830897 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.830873    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"]
Sep 17 16:58:11.832981 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.832967    2577 scope.go:117] "RemoveContainer" containerID="3fb3ad7e7f459eb05cd9b068aa70df5e5f279761cf07cb15f5fe691f9e6f7e44"
Sep 17 16:58:11.833288 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:58:11.833267    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb3ad7e7f459eb05cd9b068aa70df5e5f279761cf07cb15f5fe691f9e6f7e44\": container with ID starting with 3fb3ad7e7f459eb05cd9b068aa70df5e5f279761cf07cb15f5fe691f9e6f7e44 not found: ID does not exist" containerID="3fb3ad7e7f459eb05cd9b068aa70df5e5f279761cf07cb15f5fe691f9e6f7e44"
Sep 17 16:58:11.833377 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.833300    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb3ad7e7f459eb05cd9b068aa70df5e5f279761cf07cb15f5fe691f9e6f7e44"} err="failed to get container status \"3fb3ad7e7f459eb05cd9b068aa70df5e5f279761cf07cb15f5fe691f9e6f7e44\": rpc error: code = NotFound desc = could not find container \"3fb3ad7e7f459eb05cd9b068aa70df5e5f279761cf07cb15f5fe691f9e6f7e44\": container with ID starting with 3fb3ad7e7f459eb05cd9b068aa70df5e5f279761cf07cb15f5fe691f9e6f7e44 not found: ID does not exist"
Sep 17 16:58:11.833377 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.833322    2577 scope.go:117] "RemoveContainer" containerID="54fd0385d77f364c22ccb194135ac038eed52141a833fa5496730993ebd813c5"
Sep 17 16:58:11.833567 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:58:11.833547    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54fd0385d77f364c22ccb194135ac038eed52141a833fa5496730993ebd813c5\": container with ID starting with 54fd0385d77f364c22ccb194135ac038eed52141a833fa5496730993ebd813c5 not found: ID does not exist" containerID="54fd0385d77f364c22ccb194135ac038eed52141a833fa5496730993ebd813c5"
Sep 17 16:58:11.833609 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.833574    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54fd0385d77f364c22ccb194135ac038eed52141a833fa5496730993ebd813c5"} err="failed to get container status \"54fd0385d77f364c22ccb194135ac038eed52141a833fa5496730993ebd813c5\": rpc error: code = NotFound desc = could not find container \"54fd0385d77f364c22ccb194135ac038eed52141a833fa5496730993ebd813c5\": container with ID starting with 54fd0385d77f364c22ccb194135ac038eed52141a833fa5496730993ebd813c5 not found: ID does not exist"
Sep 17 16:58:11.833609 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.833590    2577 scope.go:117] "RemoveContainer" containerID="7f200848c658f82983561d69f5f67dcf4ca3783311dccf1c729cd7203d414cbf"
Sep 17 16:58:11.833824 ip-10-0-135-124 kubenswrapper[2577]: E0917 16:58:11.833807    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f200848c658f82983561d69f5f67dcf4ca3783311dccf1c729cd7203d414cbf\": container with ID starting with 7f200848c658f82983561d69f5f67dcf4ca3783311dccf1c729cd7203d414cbf not found: ID does not exist" containerID="7f200848c658f82983561d69f5f67dcf4ca3783311dccf1c729cd7203d414cbf"
Sep 17 16:58:11.833876 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.833833    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f200848c658f82983561d69f5f67dcf4ca3783311dccf1c729cd7203d414cbf"} err="failed to get container status \"7f200848c658f82983561d69f5f67dcf4ca3783311dccf1c729cd7203d414cbf\": rpc error: code = NotFound desc = could not find container \"7f200848c658f82983561d69f5f67dcf4ca3783311dccf1c729cd7203d414cbf\": container with ID starting with 7f200848c658f82983561d69f5f67dcf4ca3783311dccf1c729cd7203d414cbf not found: ID does not exist"
Sep 17 16:58:11.836958 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:11.836937    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dr5gs"]
Sep 17 16:58:12.447836 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:58:12.447795    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" path="/var/lib/kubelet/pods/feebd8bc-0adc-43a9-accd-ba3024bd19ea/volumes"
Sep 17 16:59:52.302885 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:59:52.302846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz" event={"ID":"403e2840-08bb-40f5-a5d9-d9089cbaefb1","Type":"ContainerStarted","Data":"6d8767cbab0664c74c011ec02acecbc9ed7aab02c18553150db120843dfd8a3e"}
Sep 17 16:59:52.303373 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:59:52.302913    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz" event={"ID":"403e2840-08bb-40f5-a5d9-d9089cbaefb1","Type":"ContainerStarted","Data":"3f2dc4555a633b84b940061dd2fbbc084466dab32714b0dd5b3240cf065fc9c3"}
Sep 17 16:59:52.303373 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:59:52.303024    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:59:52.322617 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:59:52.322564    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz" podStartSLOduration=2.7249521039999998 podStartE2EDuration="2m13.322544731s" totalImagesPullingTime="2m5.649836753s" totalInitContainerRuntime="4.947755874s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:59:52.320984146 +0000 UTC m=+1110.345892433" watchObservedRunningTime="2026-09-17 16:59:52.322544731 +0000 UTC m=+1110.347453065"
Sep 17 16:59:53.307468 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:59:53.307431    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:59:53.308848 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:59:53.308817    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz" podUID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:59:54.311509 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:59:54.311458    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz" podUID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:59:59.315991 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:59:59.315961    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 16:59:59.316514 ip-10-0-135-124 kubenswrapper[2577]: I0917 16:59:59.316453    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz" podUID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:00:09.317230 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:09.317196    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 17:00:25.840870 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:25.840835    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"]
Sep 17 17:00:25.841390 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:25.841084    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz" podUID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerName="kserve-container" containerID="cri-o://3f2dc4555a633b84b940061dd2fbbc084466dab32714b0dd5b3240cf065fc9c3" gracePeriod=30
Sep 17 17:00:25.841390 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:25.841106    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz" podUID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerName="kube-rbac-proxy" containerID="cri-o://6d8767cbab0664c74c011ec02acecbc9ed7aab02c18553150db120843dfd8a3e" gracePeriod=30
Sep 17 17:00:26.453110 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:26.453073    2577 generic.go:356] "Generic (PLEG): container finished" podID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerID="6d8767cbab0664c74c011ec02acecbc9ed7aab02c18553150db120843dfd8a3e" exitCode=2
Sep 17 17:00:26.453327 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:26.453131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz" event={"ID":"403e2840-08bb-40f5-a5d9-d9089cbaefb1","Type":"ContainerDied","Data":"6d8767cbab0664c74c011ec02acecbc9ed7aab02c18553150db120843dfd8a3e"}
Sep 17 17:00:27.216984 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.216957    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 17:00:27.407203 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.407173    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403e2840-08bb-40f5-a5d9-d9089cbaefb1-proxy-tls\") pod \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") "
Sep 17 17:00:27.407380 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.407217    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kserve-provision-location\") pod \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") "
Sep 17 17:00:27.407380 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.407252    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kube-rbac-proxy-sar-config\") pod \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") "
Sep 17 17:00:27.407380 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.407274    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/403e2840-08bb-40f5-a5d9-d9089cbaefb1-cabundle-cert\") pod \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") "
Sep 17 17:00:27.407380 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.407304    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhnp5\" (UniqueName: \"kubernetes.io/projected/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kube-api-access-mhnp5\") pod \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\" (UID: \"403e2840-08bb-40f5-a5d9-d9089cbaefb1\") "
Sep 17 17:00:27.407674 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.407649    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403e2840-08bb-40f5-a5d9-d9089cbaefb1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "403e2840-08bb-40f5-a5d9-d9089cbaefb1" (UID: "403e2840-08bb-40f5-a5d9-d9089cbaefb1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:00:27.407731 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.407657    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "403e2840-08bb-40f5-a5d9-d9089cbaefb1" (UID: "403e2840-08bb-40f5-a5d9-d9089cbaefb1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:00:27.409402 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.409379    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403e2840-08bb-40f5-a5d9-d9089cbaefb1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "403e2840-08bb-40f5-a5d9-d9089cbaefb1" (UID: "403e2840-08bb-40f5-a5d9-d9089cbaefb1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:00:27.409551 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.409530    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kube-api-access-mhnp5" (OuterVolumeSpecName: "kube-api-access-mhnp5") pod "403e2840-08bb-40f5-a5d9-d9089cbaefb1" (UID: "403e2840-08bb-40f5-a5d9-d9089cbaefb1"). InnerVolumeSpecName "kube-api-access-mhnp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:00:27.410724 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.410704    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "403e2840-08bb-40f5-a5d9-d9089cbaefb1" (UID: "403e2840-08bb-40f5-a5d9-d9089cbaefb1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:00:27.460130 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.460097    2577 generic.go:356] "Generic (PLEG): container finished" podID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerID="3f2dc4555a633b84b940061dd2fbbc084466dab32714b0dd5b3240cf065fc9c3" exitCode=0
Sep 17 17:00:27.460256 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.460183    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"
Sep 17 17:00:27.460256 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.460194    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz" event={"ID":"403e2840-08bb-40f5-a5d9-d9089cbaefb1","Type":"ContainerDied","Data":"3f2dc4555a633b84b940061dd2fbbc084466dab32714b0dd5b3240cf065fc9c3"}
Sep 17 17:00:27.460256 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.460252    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz" event={"ID":"403e2840-08bb-40f5-a5d9-d9089cbaefb1","Type":"ContainerDied","Data":"6d9661df17b7aa41854457e06eaba3570e77cb4e1ba52165168cee2cb8e30d99"}
Sep 17 17:00:27.460406 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.460280    2577 scope.go:117] "RemoveContainer" containerID="6d8767cbab0664c74c011ec02acecbc9ed7aab02c18553150db120843dfd8a3e"
Sep 17 17:00:27.468712 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.468692    2577 scope.go:117] "RemoveContainer" containerID="3f2dc4555a633b84b940061dd2fbbc084466dab32714b0dd5b3240cf065fc9c3"
Sep 17 17:00:27.476045 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.476027    2577 scope.go:117] "RemoveContainer" containerID="3cd8b98f52cf775fecb4c04fe99b7e6a1fb1a8002a7fba40a37dfec0d0ee2e05"
Sep 17 17:00:27.482273 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.482250    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"]
Sep 17 17:00:27.483867 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.483851    2577 scope.go:117] "RemoveContainer" containerID="6d8767cbab0664c74c011ec02acecbc9ed7aab02c18553150db120843dfd8a3e"
Sep 17 17:00:27.484118 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:00:27.484102    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d8767cbab0664c74c011ec02acecbc9ed7aab02c18553150db120843dfd8a3e\": container with ID starting with 6d8767cbab0664c74c011ec02acecbc9ed7aab02c18553150db120843dfd8a3e not found: ID does not exist" containerID="6d8767cbab0664c74c011ec02acecbc9ed7aab02c18553150db120843dfd8a3e"
Sep 17 17:00:27.484188 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.484124    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8767cbab0664c74c011ec02acecbc9ed7aab02c18553150db120843dfd8a3e"} err="failed to get container status \"6d8767cbab0664c74c011ec02acecbc9ed7aab02c18553150db120843dfd8a3e\": rpc error: code = NotFound desc = could not find container \"6d8767cbab0664c74c011ec02acecbc9ed7aab02c18553150db120843dfd8a3e\": container with ID starting with 6d8767cbab0664c74c011ec02acecbc9ed7aab02c18553150db120843dfd8a3e not found: ID does not exist"
Sep 17 17:00:27.484188 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.484139    2577 scope.go:117] "RemoveContainer" containerID="3f2dc4555a633b84b940061dd2fbbc084466dab32714b0dd5b3240cf065fc9c3"
Sep 17 17:00:27.484387 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:00:27.484368    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2dc4555a633b84b940061dd2fbbc084466dab32714b0dd5b3240cf065fc9c3\": container with ID starting with 3f2dc4555a633b84b940061dd2fbbc084466dab32714b0dd5b3240cf065fc9c3 not found: ID does not exist" containerID="3f2dc4555a633b84b940061dd2fbbc084466dab32714b0dd5b3240cf065fc9c3"
Sep 17 17:00:27.484458 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.484398    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2dc4555a633b84b940061dd2fbbc084466dab32714b0dd5b3240cf065fc9c3"} err="failed to get container status \"3f2dc4555a633b84b940061dd2fbbc084466dab32714b0dd5b3240cf065fc9c3\": rpc error: code = NotFound desc = could not find container \"3f2dc4555a633b84b940061dd2fbbc084466dab32714b0dd5b3240cf065fc9c3\": container with ID starting with 3f2dc4555a633b84b940061dd2fbbc084466dab32714b0dd5b3240cf065fc9c3 not found: ID does not exist"
Sep 17 17:00:27.484458 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.484422    2577 scope.go:117] "RemoveContainer" containerID="3cd8b98f52cf775fecb4c04fe99b7e6a1fb1a8002a7fba40a37dfec0d0ee2e05"
Sep 17 17:00:27.484692 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:00:27.484675    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cd8b98f52cf775fecb4c04fe99b7e6a1fb1a8002a7fba40a37dfec0d0ee2e05\": container with ID starting with 3cd8b98f52cf775fecb4c04fe99b7e6a1fb1a8002a7fba40a37dfec0d0ee2e05 not found: ID does not exist" containerID="3cd8b98f52cf775fecb4c04fe99b7e6a1fb1a8002a7fba40a37dfec0d0ee2e05"
Sep 17 17:00:27.484733 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.484697    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd8b98f52cf775fecb4c04fe99b7e6a1fb1a8002a7fba40a37dfec0d0ee2e05"} err="failed to get container status \"3cd8b98f52cf775fecb4c04fe99b7e6a1fb1a8002a7fba40a37dfec0d0ee2e05\": rpc error: code = NotFound desc = could not find container \"3cd8b98f52cf775fecb4c04fe99b7e6a1fb1a8002a7fba40a37dfec0d0ee2e05\": container with ID starting with 3cd8b98f52cf775fecb4c04fe99b7e6a1fb1a8002a7fba40a37dfec0d0ee2e05 not found: ID does not exist"
Sep 17 17:00:27.487736 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.487717    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-sjzdz"]
Sep 17 17:00:27.507922 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.507901    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/403e2840-08bb-40f5-a5d9-d9089cbaefb1-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:00:27.507922 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.507919    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:00:27.508021 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.507928    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:00:27.508021 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.507938    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/403e2840-08bb-40f5-a5d9-d9089cbaefb1-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:00:27.508021 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:27.507947    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhnp5\" (UniqueName: \"kubernetes.io/projected/403e2840-08bb-40f5-a5d9-d9089cbaefb1-kube-api-access-mhnp5\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:00:28.447179 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:28.447127    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" path="/var/lib/kubelet/pods/403e2840-08bb-40f5-a5d9-d9089cbaefb1/volumes"
Sep 17 17:00:31.019977 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.019941    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"]
Sep 17 17:00:31.020365 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020311    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerName="storage-initializer"
Sep 17 17:00:31.020365 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020325    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerName="storage-initializer"
Sep 17 17:00:31.020365 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020336    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kserve-container"
Sep 17 17:00:31.020365 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020341    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kserve-container"
Sep 17 17:00:31.020365 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020350    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kube-rbac-proxy"
Sep 17 17:00:31.020365 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020356    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kube-rbac-proxy"
Sep 17 17:00:31.020365 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020363    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerName="kube-rbac-proxy"
Sep 17 17:00:31.020365 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020368    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerName="kube-rbac-proxy"
Sep 17 17:00:31.020599 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020384    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="storage-initializer"
Sep 17 17:00:31.020599 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020390    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="storage-initializer"
Sep 17 17:00:31.020599 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020396    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerName="kserve-container"
Sep 17 17:00:31.020599 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020401    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerName="kserve-container"
Sep 17 17:00:31.020599 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020456    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerName="kube-rbac-proxy"
Sep 17 17:00:31.020599 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020465    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kube-rbac-proxy"
Sep 17 17:00:31.020599 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020471    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="feebd8bc-0adc-43a9-accd-ba3024bd19ea" containerName="kserve-container"
Sep 17 17:00:31.020599 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.020478    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="403e2840-08bb-40f5-a5d9-d9089cbaefb1" containerName="kserve-container"
Sep 17 17:00:31.023385 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.023369    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.025966 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.025945    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:00:31.026101 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.026086    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 17 17:00:31.026176 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.026097    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:00:31.026176 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.026095    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 17 17:00:31.026375 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.026359    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:00:31.026462 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.026447    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:00:31.027099 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.027087    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:00:31.032501 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.032469    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"]
Sep 17 17:00:31.135435 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.135398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89b20636-5ab8-450d-a2da-dc2a49887d23-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-47dn6\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.135435 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.135440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b20636-5ab8-450d-a2da-dc2a49887d23-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-47dn6\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.135654 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.135458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t4xx\" (UniqueName: \"kubernetes.io/projected/89b20636-5ab8-450d-a2da-dc2a49887d23-kube-api-access-9t4xx\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-47dn6\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.135654 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.135559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89b20636-5ab8-450d-a2da-dc2a49887d23-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-47dn6\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.135654 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.135607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89b20636-5ab8-450d-a2da-dc2a49887d23-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-47dn6\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.236756 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.236723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b20636-5ab8-450d-a2da-dc2a49887d23-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-47dn6\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.236756 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.236757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t4xx\" (UniqueName: \"kubernetes.io/projected/89b20636-5ab8-450d-a2da-dc2a49887d23-kube-api-access-9t4xx\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-47dn6\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.236962 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.236798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89b20636-5ab8-450d-a2da-dc2a49887d23-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-47dn6\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.236962 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.236832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89b20636-5ab8-450d-a2da-dc2a49887d23-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-47dn6\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.236962 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.236868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89b20636-5ab8-450d-a2da-dc2a49887d23-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-47dn6\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.237365 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.237340    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89b20636-5ab8-450d-a2da-dc2a49887d23-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-47dn6\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.237571 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.237547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89b20636-5ab8-450d-a2da-dc2a49887d23-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-47dn6\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.237665 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.237573    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89b20636-5ab8-450d-a2da-dc2a49887d23-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-47dn6\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.239301 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.239283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b20636-5ab8-450d-a2da-dc2a49887d23-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-47dn6\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.245128 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.245104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t4xx\" (UniqueName: \"kubernetes.io/projected/89b20636-5ab8-450d-a2da-dc2a49887d23-kube-api-access-9t4xx\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-47dn6\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.334561 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.334522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:31.605801 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:31.605731    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"]
Sep 17 17:00:31.608869 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:00:31.608833    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b20636_5ab8_450d_a2da_dc2a49887d23.slice/crio-090ee66dacbe8e373457d5e8dd6b60ac15a698e65d4f33b67ebaeed7fde1be5e WatchSource:0}: Error finding container 090ee66dacbe8e373457d5e8dd6b60ac15a698e65d4f33b67ebaeed7fde1be5e: Status 404 returned error can't find the container with id 090ee66dacbe8e373457d5e8dd6b60ac15a698e65d4f33b67ebaeed7fde1be5e
Sep 17 17:00:32.482562 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:32.482525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6" event={"ID":"89b20636-5ab8-450d-a2da-dc2a49887d23","Type":"ContainerStarted","Data":"fa0526ec9d17e94249558877b9442bbd369d7a850b9631a5cc3330cc4c95a2b4"}
Sep 17 17:00:32.482941 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:32.482577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6" event={"ID":"89b20636-5ab8-450d-a2da-dc2a49887d23","Type":"ContainerStarted","Data":"090ee66dacbe8e373457d5e8dd6b60ac15a698e65d4f33b67ebaeed7fde1be5e"}
Sep 17 17:00:36.500749 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:36.500719    2577 generic.go:356] "Generic (PLEG): container finished" podID="89b20636-5ab8-450d-a2da-dc2a49887d23" containerID="fa0526ec9d17e94249558877b9442bbd369d7a850b9631a5cc3330cc4c95a2b4" exitCode=0
Sep 17 17:00:36.501072 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:36.500789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6" event={"ID":"89b20636-5ab8-450d-a2da-dc2a49887d23","Type":"ContainerDied","Data":"fa0526ec9d17e94249558877b9442bbd369d7a850b9631a5cc3330cc4c95a2b4"}
Sep 17 17:00:37.507733 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:37.507693    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6" event={"ID":"89b20636-5ab8-450d-a2da-dc2a49887d23","Type":"ContainerStarted","Data":"803b8cfb4f764493b4f131633a311a67fe7074444a7a63c3682d208289033b6f"}
Sep 17 17:00:37.508126 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:37.507776    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6" event={"ID":"89b20636-5ab8-450d-a2da-dc2a49887d23","Type":"ContainerStarted","Data":"52258098346db646d94503d12e1fb1c2bc9388db91b33c1ad0d55e52df0bdb90"}
Sep 17 17:00:37.508222 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:37.508182    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:37.508276 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:37.508256    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:00:37.527821 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:37.527771    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6" podStartSLOduration=1.727802478 podStartE2EDuration="6.527757148s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.79995467s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:00:37.525680963 +0000 UTC m=+1155.550589257" watchObservedRunningTime="2026-09-17 17:00:37.527757148 +0000 UTC m=+1155.552665436"
Sep 17 17:00:43.518268 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:00:43.518235    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:01:22.472225 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:01:22.472195    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 17:01:22.472732 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:01:22.472605    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 17:02:08.482597 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:08.482471    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:02:17.030951 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:17.030863    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"]
Sep 17 17:02:17.031627 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:17.031086    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6" podUID="89b20636-5ab8-450d-a2da-dc2a49887d23" containerName="kserve-container" containerID="cri-o://52258098346db646d94503d12e1fb1c2bc9388db91b33c1ad0d55e52df0bdb90" gracePeriod=30
Sep 17 17:02:17.031627 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:17.031114    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6" podUID="89b20636-5ab8-450d-a2da-dc2a49887d23" containerName="kube-rbac-proxy" containerID="cri-o://803b8cfb4f764493b4f131633a311a67fe7074444a7a63c3682d208289033b6f" gracePeriod=30
Sep 17 17:02:17.953335 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:17.953285    2577 generic.go:356] "Generic (PLEG): container finished" podID="89b20636-5ab8-450d-a2da-dc2a49887d23" containerID="803b8cfb4f764493b4f131633a311a67fe7074444a7a63c3682d208289033b6f" exitCode=2
Sep 17 17:02:17.953500 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:17.953343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6" event={"ID":"89b20636-5ab8-450d-a2da-dc2a49887d23","Type":"ContainerDied","Data":"803b8cfb4f764493b4f131633a311a67fe7074444a7a63c3682d208289033b6f"}
Sep 17 17:02:18.371273 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.371242    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:02:18.441856 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.441829    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89b20636-5ab8-450d-a2da-dc2a49887d23-kube-rbac-proxy-sar-config\") pod \"89b20636-5ab8-450d-a2da-dc2a49887d23\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") "
Sep 17 17:02:18.441980 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.441878    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t4xx\" (UniqueName: \"kubernetes.io/projected/89b20636-5ab8-450d-a2da-dc2a49887d23-kube-api-access-9t4xx\") pod \"89b20636-5ab8-450d-a2da-dc2a49887d23\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") "
Sep 17 17:02:18.441980 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.441929    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b20636-5ab8-450d-a2da-dc2a49887d23-proxy-tls\") pod \"89b20636-5ab8-450d-a2da-dc2a49887d23\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") "
Sep 17 17:02:18.441980 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.441966    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89b20636-5ab8-450d-a2da-dc2a49887d23-cabundle-cert\") pod \"89b20636-5ab8-450d-a2da-dc2a49887d23\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") "
Sep 17 17:02:18.442111 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.442009    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89b20636-5ab8-450d-a2da-dc2a49887d23-kserve-provision-location\") pod \"89b20636-5ab8-450d-a2da-dc2a49887d23\" (UID: \"89b20636-5ab8-450d-a2da-dc2a49887d23\") "
Sep 17 17:02:18.442220 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.442194    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b20636-5ab8-450d-a2da-dc2a49887d23-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89b20636-5ab8-450d-a2da-dc2a49887d23" (UID: "89b20636-5ab8-450d-a2da-dc2a49887d23"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:02:18.442483 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.442446    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b20636-5ab8-450d-a2da-dc2a49887d23-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89b20636-5ab8-450d-a2da-dc2a49887d23" (UID: "89b20636-5ab8-450d-a2da-dc2a49887d23"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:02:18.444039 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.444009    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b20636-5ab8-450d-a2da-dc2a49887d23-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89b20636-5ab8-450d-a2da-dc2a49887d23" (UID: "89b20636-5ab8-450d-a2da-dc2a49887d23"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:02:18.444127 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.444057    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b20636-5ab8-450d-a2da-dc2a49887d23-kube-api-access-9t4xx" (OuterVolumeSpecName: "kube-api-access-9t4xx") pod "89b20636-5ab8-450d-a2da-dc2a49887d23" (UID: "89b20636-5ab8-450d-a2da-dc2a49887d23"). InnerVolumeSpecName "kube-api-access-9t4xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:02:18.447586 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.447556    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b20636-5ab8-450d-a2da-dc2a49887d23-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89b20636-5ab8-450d-a2da-dc2a49887d23" (UID: "89b20636-5ab8-450d-a2da-dc2a49887d23"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:18.543333 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.543315    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b20636-5ab8-450d-a2da-dc2a49887d23-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:02:18.543333 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.543334    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89b20636-5ab8-450d-a2da-dc2a49887d23-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:02:18.543491 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.543344    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89b20636-5ab8-450d-a2da-dc2a49887d23-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:02:18.543491 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.543354    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89b20636-5ab8-450d-a2da-dc2a49887d23-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:02:18.543491 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.543362    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t4xx\" (UniqueName: \"kubernetes.io/projected/89b20636-5ab8-450d-a2da-dc2a49887d23-kube-api-access-9t4xx\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:02:18.959360 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.959274    2577 generic.go:356] "Generic (PLEG): container finished" podID="89b20636-5ab8-450d-a2da-dc2a49887d23" containerID="52258098346db646d94503d12e1fb1c2bc9388db91b33c1ad0d55e52df0bdb90" exitCode=0
Sep 17 17:02:18.959360 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.959352    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"
Sep 17 17:02:18.959581 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.959365    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6" event={"ID":"89b20636-5ab8-450d-a2da-dc2a49887d23","Type":"ContainerDied","Data":"52258098346db646d94503d12e1fb1c2bc9388db91b33c1ad0d55e52df0bdb90"}
Sep 17 17:02:18.959581 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.959444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6" event={"ID":"89b20636-5ab8-450d-a2da-dc2a49887d23","Type":"ContainerDied","Data":"090ee66dacbe8e373457d5e8dd6b60ac15a698e65d4f33b67ebaeed7fde1be5e"}
Sep 17 17:02:18.959581 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.959484    2577 scope.go:117] "RemoveContainer" containerID="803b8cfb4f764493b4f131633a311a67fe7074444a7a63c3682d208289033b6f"
Sep 17 17:02:18.967911 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.967892    2577 scope.go:117] "RemoveContainer" containerID="52258098346db646d94503d12e1fb1c2bc9388db91b33c1ad0d55e52df0bdb90"
Sep 17 17:02:18.975282 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.975265    2577 scope.go:117] "RemoveContainer" containerID="fa0526ec9d17e94249558877b9442bbd369d7a850b9631a5cc3330cc4c95a2b4"
Sep 17 17:02:18.980173 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.980140    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"]
Sep 17 17:02:18.982767 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.982749    2577 scope.go:117] "RemoveContainer" containerID="803b8cfb4f764493b4f131633a311a67fe7074444a7a63c3682d208289033b6f"
Sep 17 17:02:18.983046 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:02:18.983027    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"803b8cfb4f764493b4f131633a311a67fe7074444a7a63c3682d208289033b6f\": container with ID starting with 803b8cfb4f764493b4f131633a311a67fe7074444a7a63c3682d208289033b6f not found: ID does not exist" containerID="803b8cfb4f764493b4f131633a311a67fe7074444a7a63c3682d208289033b6f"
Sep 17 17:02:18.983091 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.983054    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803b8cfb4f764493b4f131633a311a67fe7074444a7a63c3682d208289033b6f"} err="failed to get container status \"803b8cfb4f764493b4f131633a311a67fe7074444a7a63c3682d208289033b6f\": rpc error: code = NotFound desc = could not find container \"803b8cfb4f764493b4f131633a311a67fe7074444a7a63c3682d208289033b6f\": container with ID starting with 803b8cfb4f764493b4f131633a311a67fe7074444a7a63c3682d208289033b6f not found: ID does not exist"
Sep 17 17:02:18.983091 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.983071    2577 scope.go:117] "RemoveContainer" containerID="52258098346db646d94503d12e1fb1c2bc9388db91b33c1ad0d55e52df0bdb90"
Sep 17 17:02:18.983355 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:02:18.983337    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52258098346db646d94503d12e1fb1c2bc9388db91b33c1ad0d55e52df0bdb90\": container with ID starting with 52258098346db646d94503d12e1fb1c2bc9388db91b33c1ad0d55e52df0bdb90 not found: ID does not exist" containerID="52258098346db646d94503d12e1fb1c2bc9388db91b33c1ad0d55e52df0bdb90"
Sep 17 17:02:18.983425 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.983364    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52258098346db646d94503d12e1fb1c2bc9388db91b33c1ad0d55e52df0bdb90"} err="failed to get container status \"52258098346db646d94503d12e1fb1c2bc9388db91b33c1ad0d55e52df0bdb90\": rpc error: code = NotFound desc = could not find container \"52258098346db646d94503d12e1fb1c2bc9388db91b33c1ad0d55e52df0bdb90\": container with ID starting with 52258098346db646d94503d12e1fb1c2bc9388db91b33c1ad0d55e52df0bdb90 not found: ID does not exist"
Sep 17 17:02:18.983425 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.983385    2577 scope.go:117] "RemoveContainer" containerID="fa0526ec9d17e94249558877b9442bbd369d7a850b9631a5cc3330cc4c95a2b4"
Sep 17 17:02:18.983638 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:02:18.983621    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0526ec9d17e94249558877b9442bbd369d7a850b9631a5cc3330cc4c95a2b4\": container with ID starting with fa0526ec9d17e94249558877b9442bbd369d7a850b9631a5cc3330cc4c95a2b4 not found: ID does not exist" containerID="fa0526ec9d17e94249558877b9442bbd369d7a850b9631a5cc3330cc4c95a2b4"
Sep 17 17:02:18.983676 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.983643    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0526ec9d17e94249558877b9442bbd369d7a850b9631a5cc3330cc4c95a2b4"} err="failed to get container status \"fa0526ec9d17e94249558877b9442bbd369d7a850b9631a5cc3330cc4c95a2b4\": rpc error: code = NotFound desc = could not find container \"fa0526ec9d17e94249558877b9442bbd369d7a850b9631a5cc3330cc4c95a2b4\": container with ID starting with fa0526ec9d17e94249558877b9442bbd369d7a850b9631a5cc3330cc4c95a2b4 not found: ID does not exist"
Sep 17 17:02:18.983962 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:18.983946    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-47dn6"]
Sep 17 17:02:20.447072 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:20.447043    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89b20636-5ab8-450d-a2da-dc2a49887d23" path="/var/lib/kubelet/pods/89b20636-5ab8-450d-a2da-dc2a49887d23/volumes"
Sep 17 17:02:22.213800 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.213768    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"]
Sep 17 17:02:22.214187 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.214072    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b20636-5ab8-450d-a2da-dc2a49887d23" containerName="storage-initializer"
Sep 17 17:02:22.214187 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.214083    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b20636-5ab8-450d-a2da-dc2a49887d23" containerName="storage-initializer"
Sep 17 17:02:22.214187 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.214091    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b20636-5ab8-450d-a2da-dc2a49887d23" containerName="kserve-container"
Sep 17 17:02:22.214187 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.214096    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b20636-5ab8-450d-a2da-dc2a49887d23" containerName="kserve-container"
Sep 17 17:02:22.214187 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.214105    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b20636-5ab8-450d-a2da-dc2a49887d23" containerName="kube-rbac-proxy"
Sep 17 17:02:22.214187 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.214111    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b20636-5ab8-450d-a2da-dc2a49887d23" containerName="kube-rbac-proxy"
Sep 17 17:02:22.214187 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.214179    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89b20636-5ab8-450d-a2da-dc2a49887d23" containerName="kserve-container"
Sep 17 17:02:22.214187 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.214191    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89b20636-5ab8-450d-a2da-dc2a49887d23" containerName="kube-rbac-proxy"
Sep 17 17:02:22.219033 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.219016    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.221761 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.221736    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 17 17:02:22.221761 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.221758    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:02:22.221942 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.221863    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:02:22.222004 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.221939    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:02:22.222004 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.221985    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:02:22.222117 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.222101    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:02:22.222906 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.222884    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 17 17:02:22.227442 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.227418    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"]
Sep 17 17:02:22.271395 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.271370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8f22955-ab48-456b-8fc6-16158e357214-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.271488 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.271405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dws9l\" (UniqueName: \"kubernetes.io/projected/f8f22955-ab48-456b-8fc6-16158e357214-kube-api-access-dws9l\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.271488 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.271424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8f22955-ab48-456b-8fc6-16158e357214-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.271488 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.271450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8f22955-ab48-456b-8fc6-16158e357214-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.271488 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.271470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8f22955-ab48-456b-8fc6-16158e357214-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.372090 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.372059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8f22955-ab48-456b-8fc6-16158e357214-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.372223 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.372099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dws9l\" (UniqueName: \"kubernetes.io/projected/f8f22955-ab48-456b-8fc6-16158e357214-kube-api-access-dws9l\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.372223 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.372120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8f22955-ab48-456b-8fc6-16158e357214-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.372223 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.372148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8f22955-ab48-456b-8fc6-16158e357214-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.372223 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.372199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8f22955-ab48-456b-8fc6-16158e357214-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.372575 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.372555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8f22955-ab48-456b-8fc6-16158e357214-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.374462 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.374443    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 17 17:02:22.374541 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.374446    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 17 17:02:22.374725 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.374712    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:02:22.379835 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.379819    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:02:22.382733 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:02:22.382718    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 17 17:02:22.382820 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:02:22.382788    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8f22955-ab48-456b-8fc6-16158e357214-proxy-tls podName:f8f22955-ab48-456b-8fc6-16158e357214 nodeName:}" failed. No retries permitted until 2026-09-17 17:02:22.882768385 +0000 UTC m=+1260.907676657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f8f22955-ab48-456b-8fc6-16158e357214-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6" (UID: "f8f22955-ab48-456b-8fc6-16158e357214") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 17 17:02:22.383205 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.383188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8f22955-ab48-456b-8fc6-16158e357214-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.383272 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.383256    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8f22955-ab48-456b-8fc6-16158e357214-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.390351 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.390338    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:02:22.400922 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.400899    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dws9l\" (UniqueName: \"kubernetes.io/projected/f8f22955-ab48-456b-8fc6-16158e357214-kube-api-access-dws9l\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.978094 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.978063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8f22955-ab48-456b-8fc6-16158e357214-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:22.980409 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:22.980380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8f22955-ab48-456b-8fc6-16158e357214-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:23.132608 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:23.132580    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:02:23.140456 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:23.140437    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:23.414306 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:23.414271    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"]
Sep 17 17:02:23.416920 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:02:23.416897    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f22955_ab48_456b_8fc6_16158e357214.slice/crio-7cd28373a04917de877453a075a9de5a5f0e35c40a984933496033c40bf82c45 WatchSource:0}: Error finding container 7cd28373a04917de877453a075a9de5a5f0e35c40a984933496033c40bf82c45: Status 404 returned error can't find the container with id 7cd28373a04917de877453a075a9de5a5f0e35c40a984933496033c40bf82c45
Sep 17 17:02:23.422292 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:23.422271    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:02:23.986123 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:23.986085    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6" event={"ID":"f8f22955-ab48-456b-8fc6-16158e357214","Type":"ContainerStarted","Data":"9b6c02ee659c164bc5c124dba872e90169b3bed799becd34ab957a0e778648f5"}
Sep 17 17:02:23.986329 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:23.986155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6" event={"ID":"f8f22955-ab48-456b-8fc6-16158e357214","Type":"ContainerStarted","Data":"7cd28373a04917de877453a075a9de5a5f0e35c40a984933496033c40bf82c45"}
Sep 17 17:02:44.078295 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:44.078260    2577 generic.go:356] "Generic (PLEG): container finished" podID="f8f22955-ab48-456b-8fc6-16158e357214" containerID="9b6c02ee659c164bc5c124dba872e90169b3bed799becd34ab957a0e778648f5" exitCode=0
Sep 17 17:02:44.078685 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:44.078334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6" event={"ID":"f8f22955-ab48-456b-8fc6-16158e357214","Type":"ContainerDied","Data":"9b6c02ee659c164bc5c124dba872e90169b3bed799becd34ab957a0e778648f5"}
Sep 17 17:02:45.084634 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:45.084596    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6" event={"ID":"f8f22955-ab48-456b-8fc6-16158e357214","Type":"ContainerStarted","Data":"79bf42737725d319ab540730f6002bbe5f795c116ad6dd4fb4bea67e940c82ee"}
Sep 17 17:02:45.085093 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:45.084668    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6" event={"ID":"f8f22955-ab48-456b-8fc6-16158e357214","Type":"ContainerStarted","Data":"420101cb3f9907419fb8c762a36027f312df1910589ea445d2eb5e766244566e"}
Sep 17 17:02:45.085093 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:45.084877    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:45.105259 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:45.105213    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6" podStartSLOduration=2.712743686 podStartE2EDuration="23.105199545s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.392455859s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:02:45.103794372 +0000 UTC m=+1283.128702671" watchObservedRunningTime="2026-09-17 17:02:45.105199545 +0000 UTC m=+1283.130107900"
Sep 17 17:02:46.089994 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:46.089962    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:02:52.099807 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:02:52.099774    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:04:19.475664 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:19.475632    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:04:28.667131 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:28.667098    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"]
Sep 17 17:04:28.667628 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:28.667338    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6" podUID="f8f22955-ab48-456b-8fc6-16158e357214" containerName="kserve-container" containerID="cri-o://420101cb3f9907419fb8c762a36027f312df1910589ea445d2eb5e766244566e" gracePeriod=30
Sep 17 17:04:28.667628 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:28.667375    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6" podUID="f8f22955-ab48-456b-8fc6-16158e357214" containerName="kube-rbac-proxy" containerID="cri-o://79bf42737725d319ab540730f6002bbe5f795c116ad6dd4fb4bea67e940c82ee" gracePeriod=30
Sep 17 17:04:29.443327 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:29.443277    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6" podUID="f8f22955-ab48-456b-8fc6-16158e357214" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-autogluon-v2-variants/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 17:04:29.556349 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:29.556314    2577 generic.go:356] "Generic (PLEG): container finished" podID="f8f22955-ab48-456b-8fc6-16158e357214" containerID="79bf42737725d319ab540730f6002bbe5f795c116ad6dd4fb4bea67e940c82ee" exitCode=2
Sep 17 17:04:29.556505 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:29.556365    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6" event={"ID":"f8f22955-ab48-456b-8fc6-16158e357214","Type":"ContainerDied","Data":"79bf42737725d319ab540730f6002bbe5f795c116ad6dd4fb4bea67e940c82ee"}
Sep 17 17:04:30.012768 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.012743    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:04:30.095118 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.095053    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8f22955-ab48-456b-8fc6-16158e357214-kube-rbac-proxy-sar-config\") pod \"f8f22955-ab48-456b-8fc6-16158e357214\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") "
Sep 17 17:04:30.095118 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.095093    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dws9l\" (UniqueName: \"kubernetes.io/projected/f8f22955-ab48-456b-8fc6-16158e357214-kube-api-access-dws9l\") pod \"f8f22955-ab48-456b-8fc6-16158e357214\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") "
Sep 17 17:04:30.095332 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.095232    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8f22955-ab48-456b-8fc6-16158e357214-proxy-tls\") pod \"f8f22955-ab48-456b-8fc6-16158e357214\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") "
Sep 17 17:04:30.095332 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.095283    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8f22955-ab48-456b-8fc6-16158e357214-kserve-provision-location\") pod \"f8f22955-ab48-456b-8fc6-16158e357214\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") "
Sep 17 17:04:30.095332 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.095301    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8f22955-ab48-456b-8fc6-16158e357214-cabundle-cert\") pod \"f8f22955-ab48-456b-8fc6-16158e357214\" (UID: \"f8f22955-ab48-456b-8fc6-16158e357214\") "
Sep 17 17:04:30.095479 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.095350    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f22955-ab48-456b-8fc6-16158e357214-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f8f22955-ab48-456b-8fc6-16158e357214" (UID: "f8f22955-ab48-456b-8fc6-16158e357214"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:04:30.095537 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.095520    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8f22955-ab48-456b-8fc6-16158e357214-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:04:30.095692 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.095672    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f22955-ab48-456b-8fc6-16158e357214-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f8f22955-ab48-456b-8fc6-16158e357214" (UID: "f8f22955-ab48-456b-8fc6-16158e357214"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:04:30.097136 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.097112    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f22955-ab48-456b-8fc6-16158e357214-kube-api-access-dws9l" (OuterVolumeSpecName: "kube-api-access-dws9l") pod "f8f22955-ab48-456b-8fc6-16158e357214" (UID: "f8f22955-ab48-456b-8fc6-16158e357214"). InnerVolumeSpecName "kube-api-access-dws9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:04:30.097246 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.097141    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8f22955-ab48-456b-8fc6-16158e357214-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f8f22955-ab48-456b-8fc6-16158e357214" (UID: "f8f22955-ab48-456b-8fc6-16158e357214"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:04:30.099619 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.099594    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f22955-ab48-456b-8fc6-16158e357214-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8f22955-ab48-456b-8fc6-16158e357214" (UID: "f8f22955-ab48-456b-8fc6-16158e357214"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:04:30.196846 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.196820    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dws9l\" (UniqueName: \"kubernetes.io/projected/f8f22955-ab48-456b-8fc6-16158e357214-kube-api-access-dws9l\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:04:30.196846 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.196844    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8f22955-ab48-456b-8fc6-16158e357214-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:04:30.196985 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.196855    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8f22955-ab48-456b-8fc6-16158e357214-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:04:30.196985 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.196864    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8f22955-ab48-456b-8fc6-16158e357214-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:04:30.562663 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.562624    2577 generic.go:356] "Generic (PLEG): container finished" podID="f8f22955-ab48-456b-8fc6-16158e357214" containerID="420101cb3f9907419fb8c762a36027f312df1910589ea445d2eb5e766244566e" exitCode=0
Sep 17 17:04:30.562792 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.562686    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"
Sep 17 17:04:30.562792 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.562709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6" event={"ID":"f8f22955-ab48-456b-8fc6-16158e357214","Type":"ContainerDied","Data":"420101cb3f9907419fb8c762a36027f312df1910589ea445d2eb5e766244566e"}
Sep 17 17:04:30.562792 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.562759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6" event={"ID":"f8f22955-ab48-456b-8fc6-16158e357214","Type":"ContainerDied","Data":"7cd28373a04917de877453a075a9de5a5f0e35c40a984933496033c40bf82c45"}
Sep 17 17:04:30.562792 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.562786    2577 scope.go:117] "RemoveContainer" containerID="79bf42737725d319ab540730f6002bbe5f795c116ad6dd4fb4bea67e940c82ee"
Sep 17 17:04:30.571553 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.571534    2577 scope.go:117] "RemoveContainer" containerID="420101cb3f9907419fb8c762a36027f312df1910589ea445d2eb5e766244566e"
Sep 17 17:04:30.579817 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.579794    2577 scope.go:117] "RemoveContainer" containerID="9b6c02ee659c164bc5c124dba872e90169b3bed799becd34ab957a0e778648f5"
Sep 17 17:04:30.581271 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.581247    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"]
Sep 17 17:04:30.584752 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.584728    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-87kb6"]
Sep 17 17:04:30.587905 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.587884    2577 scope.go:117] "RemoveContainer" containerID="79bf42737725d319ab540730f6002bbe5f795c116ad6dd4fb4bea67e940c82ee"
Sep 17 17:04:30.588206 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:04:30.588149    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79bf42737725d319ab540730f6002bbe5f795c116ad6dd4fb4bea67e940c82ee\": container with ID starting with 79bf42737725d319ab540730f6002bbe5f795c116ad6dd4fb4bea67e940c82ee not found: ID does not exist" containerID="79bf42737725d319ab540730f6002bbe5f795c116ad6dd4fb4bea67e940c82ee"
Sep 17 17:04:30.588275 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.588215    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79bf42737725d319ab540730f6002bbe5f795c116ad6dd4fb4bea67e940c82ee"} err="failed to get container status \"79bf42737725d319ab540730f6002bbe5f795c116ad6dd4fb4bea67e940c82ee\": rpc error: code = NotFound desc = could not find container \"79bf42737725d319ab540730f6002bbe5f795c116ad6dd4fb4bea67e940c82ee\": container with ID starting with 79bf42737725d319ab540730f6002bbe5f795c116ad6dd4fb4bea67e940c82ee not found: ID does not exist"
Sep 17 17:04:30.588275 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.588234    2577 scope.go:117] "RemoveContainer" containerID="420101cb3f9907419fb8c762a36027f312df1910589ea445d2eb5e766244566e"
Sep 17 17:04:30.588575 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:04:30.588556    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"420101cb3f9907419fb8c762a36027f312df1910589ea445d2eb5e766244566e\": container with ID starting with 420101cb3f9907419fb8c762a36027f312df1910589ea445d2eb5e766244566e not found: ID does not exist" containerID="420101cb3f9907419fb8c762a36027f312df1910589ea445d2eb5e766244566e"
Sep 17 17:04:30.588643 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.588584    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420101cb3f9907419fb8c762a36027f312df1910589ea445d2eb5e766244566e"} err="failed to get container status \"420101cb3f9907419fb8c762a36027f312df1910589ea445d2eb5e766244566e\": rpc error: code = NotFound desc = could not find container \"420101cb3f9907419fb8c762a36027f312df1910589ea445d2eb5e766244566e\": container with ID starting with 420101cb3f9907419fb8c762a36027f312df1910589ea445d2eb5e766244566e not found: ID does not exist"
Sep 17 17:04:30.588643 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.588607    2577 scope.go:117] "RemoveContainer" containerID="9b6c02ee659c164bc5c124dba872e90169b3bed799becd34ab957a0e778648f5"
Sep 17 17:04:30.588911 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:04:30.588892    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b6c02ee659c164bc5c124dba872e90169b3bed799becd34ab957a0e778648f5\": container with ID starting with 9b6c02ee659c164bc5c124dba872e90169b3bed799becd34ab957a0e778648f5 not found: ID does not exist" containerID="9b6c02ee659c164bc5c124dba872e90169b3bed799becd34ab957a0e778648f5"
Sep 17 17:04:30.588955 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:30.588920    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b6c02ee659c164bc5c124dba872e90169b3bed799becd34ab957a0e778648f5"} err="failed to get container status \"9b6c02ee659c164bc5c124dba872e90169b3bed799becd34ab957a0e778648f5\": rpc error: code = NotFound desc = could not find container \"9b6c02ee659c164bc5c124dba872e90169b3bed799becd34ab957a0e778648f5\": container with ID starting with 9b6c02ee659c164bc5c124dba872e90169b3bed799becd34ab957a0e778648f5 not found: ID does not exist"
Sep 17 17:04:32.446773 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:32.446727    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8f22955-ab48-456b-8fc6-16158e357214" path="/var/lib/kubelet/pods/f8f22955-ab48-456b-8fc6-16158e357214/volumes"
Sep 17 17:04:33.854654 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.854616    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"]
Sep 17 17:04:33.855020 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.854941    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8f22955-ab48-456b-8fc6-16158e357214" containerName="kube-rbac-proxy"
Sep 17 17:04:33.855020 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.854953    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8f22955-ab48-456b-8fc6-16158e357214" containerName="kube-rbac-proxy"
Sep 17 17:04:33.855020 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.854961    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8f22955-ab48-456b-8fc6-16158e357214" containerName="kserve-container"
Sep 17 17:04:33.855020 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.854967    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8f22955-ab48-456b-8fc6-16158e357214" containerName="kserve-container"
Sep 17 17:04:33.855020 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.854976    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8f22955-ab48-456b-8fc6-16158e357214" containerName="storage-initializer"
Sep 17 17:04:33.855020 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.854982    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8f22955-ab48-456b-8fc6-16158e357214" containerName="storage-initializer"
Sep 17 17:04:33.855241 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.855030    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8f22955-ab48-456b-8fc6-16158e357214" containerName="kube-rbac-proxy"
Sep 17 17:04:33.855241 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.855043    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8f22955-ab48-456b-8fc6-16158e357214" containerName="kserve-container"
Sep 17 17:04:33.859841 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.859823    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:33.862379 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.862353    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 17 17:04:33.862531 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.862489    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:04:33.862737 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.862720    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:04:33.862830 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.862798    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:04:33.863678 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.863654    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:04:33.863797 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.863692    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:04:33.863797 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.863692    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 17 17:04:33.867872 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.867848    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"]
Sep 17 17:04:33.924397 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.924370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1bcaac-435e-43f4-981b-9877fb5ece92-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:33.924487 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.924405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f1bcaac-435e-43f4-981b-9877fb5ece92-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:33.924487 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.924439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f1bcaac-435e-43f4-981b-9877fb5ece92-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:33.924487 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.924464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f1bcaac-435e-43f4-981b-9877fb5ece92-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:33.924589 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:33.924497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwqv5\" (UniqueName: \"kubernetes.io/projected/2f1bcaac-435e-43f4-981b-9877fb5ece92-kube-api-access-hwqv5\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:34.025269 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:34.025242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f1bcaac-435e-43f4-981b-9877fb5ece92-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:34.025351 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:34.025278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f1bcaac-435e-43f4-981b-9877fb5ece92-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:34.025351 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:34.025316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwqv5\" (UniqueName: \"kubernetes.io/projected/2f1bcaac-435e-43f4-981b-9877fb5ece92-kube-api-access-hwqv5\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:34.025441 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:34.025352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1bcaac-435e-43f4-981b-9877fb5ece92-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:34.025441 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:34.025377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f1bcaac-435e-43f4-981b-9877fb5ece92-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:34.025538 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:04:34.025474    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 17 17:04:34.025585 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:04:34.025546    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f1bcaac-435e-43f4-981b-9877fb5ece92-proxy-tls podName:2f1bcaac-435e-43f4-981b-9877fb5ece92 nodeName:}" failed. No retries permitted until 2026-09-17 17:04:34.525525895 +0000 UTC m=+1392.550434172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2f1bcaac-435e-43f4-981b-9877fb5ece92-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m" (UID: "2f1bcaac-435e-43f4-981b-9877fb5ece92") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 17 17:04:34.025678 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:34.025654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f1bcaac-435e-43f4-981b-9877fb5ece92-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:34.026016 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:34.025992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f1bcaac-435e-43f4-981b-9877fb5ece92-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:34.026091 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:34.026031    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f1bcaac-435e-43f4-981b-9877fb5ece92-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:34.034394 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:34.034371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwqv5\" (UniqueName: \"kubernetes.io/projected/2f1bcaac-435e-43f4-981b-9877fb5ece92-kube-api-access-hwqv5\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:34.528303 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:34.528271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1bcaac-435e-43f4-981b-9877fb5ece92-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:34.530843 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:34.530809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1bcaac-435e-43f4-981b-9877fb5ece92-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:34.771488 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:34.771451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:35.045660 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:35.045636    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"]
Sep 17 17:04:35.048090 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:04:35.048060    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1bcaac_435e_43f4_981b_9877fb5ece92.slice/crio-f5c2ff3461804aef04cecee1bfcd165829038525cb33e1b243bf1fac1a52be3f WatchSource:0}: Error finding container f5c2ff3461804aef04cecee1bfcd165829038525cb33e1b243bf1fac1a52be3f: Status 404 returned error can't find the container with id f5c2ff3461804aef04cecee1bfcd165829038525cb33e1b243bf1fac1a52be3f
Sep 17 17:04:35.594847 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:35.594805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m" event={"ID":"2f1bcaac-435e-43f4-981b-9877fb5ece92","Type":"ContainerStarted","Data":"cac1dac6e60ea282b532e8298118a7e22d36d66c335cdd7625bbcb83c413c373"}
Sep 17 17:04:35.594996 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:35.594880    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m" event={"ID":"2f1bcaac-435e-43f4-981b-9877fb5ece92","Type":"ContainerStarted","Data":"f5c2ff3461804aef04cecee1bfcd165829038525cb33e1b243bf1fac1a52be3f"}
Sep 17 17:04:40.617187 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:40.617138    2577 generic.go:356] "Generic (PLEG): container finished" podID="2f1bcaac-435e-43f4-981b-9877fb5ece92" containerID="cac1dac6e60ea282b532e8298118a7e22d36d66c335cdd7625bbcb83c413c373" exitCode=0
Sep 17 17:04:40.617568 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:40.617211    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m" event={"ID":"2f1bcaac-435e-43f4-981b-9877fb5ece92","Type":"ContainerDied","Data":"cac1dac6e60ea282b532e8298118a7e22d36d66c335cdd7625bbcb83c413c373"}
Sep 17 17:04:41.623943 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:41.623904    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m" event={"ID":"2f1bcaac-435e-43f4-981b-9877fb5ece92","Type":"ContainerStarted","Data":"0889a1ed5aaa3789d107b029524ac6adf3425fe6c99f06efeee9c31ae0a6d9d2"}
Sep 17 17:04:41.624403 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:41.623979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m" event={"ID":"2f1bcaac-435e-43f4-981b-9877fb5ece92","Type":"ContainerStarted","Data":"923b5858d8ae9bd390b110b5d292232e767a61477c150cd503fd3f193ff28414"}
Sep 17 17:04:41.624403 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:41.624196    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:41.643867 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:41.643816    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m" podStartSLOduration=3.78668036 podStartE2EDuration="8.64380201s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.85712165s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:04:41.642528951 +0000 UTC m=+1399.667437240" watchObservedRunningTime="2026-09-17 17:04:41.64380201 +0000 UTC m=+1399.668710298"
Sep 17 17:04:42.629205 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:42.629155    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:04:48.639211 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:04:48.639176    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:06:11.477236 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:11.477202    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:06:19.838268 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:19.838237    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"]
Sep 17 17:06:19.838769 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:19.838499    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m" podUID="2f1bcaac-435e-43f4-981b-9877fb5ece92" containerName="kserve-container" containerID="cri-o://923b5858d8ae9bd390b110b5d292232e767a61477c150cd503fd3f193ff28414" gracePeriod=30
Sep 17 17:06:19.838769 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:19.838513    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m" podUID="2f1bcaac-435e-43f4-981b-9877fb5ece92" containerName="kube-rbac-proxy" containerID="cri-o://0889a1ed5aaa3789d107b029524ac6adf3425fe6c99f06efeee9c31ae0a6d9d2" gracePeriod=30
Sep 17 17:06:20.053964 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:20.053925    2577 generic.go:356] "Generic (PLEG): container finished" podID="2f1bcaac-435e-43f4-981b-9877fb5ece92" containerID="0889a1ed5aaa3789d107b029524ac6adf3425fe6c99f06efeee9c31ae0a6d9d2" exitCode=2
Sep 17 17:06:20.054106 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:20.053993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m" event={"ID":"2f1bcaac-435e-43f4-981b-9877fb5ece92","Type":"ContainerDied","Data":"0889a1ed5aaa3789d107b029524ac6adf3425fe6c99f06efeee9c31ae0a6d9d2"}
Sep 17 17:06:21.060358 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.060332    2577 generic.go:356] "Generic (PLEG): container finished" podID="2f1bcaac-435e-43f4-981b-9877fb5ece92" containerID="923b5858d8ae9bd390b110b5d292232e767a61477c150cd503fd3f193ff28414" exitCode=0
Sep 17 17:06:21.060645 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.060399    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m" event={"ID":"2f1bcaac-435e-43f4-981b-9877fb5ece92","Type":"ContainerDied","Data":"923b5858d8ae9bd390b110b5d292232e767a61477c150cd503fd3f193ff28414"}
Sep 17 17:06:21.182611 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.182583    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:06:21.313672 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.313643    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f1bcaac-435e-43f4-981b-9877fb5ece92-kube-rbac-proxy-sar-config\") pod \"2f1bcaac-435e-43f4-981b-9877fb5ece92\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") "
Sep 17 17:06:21.313821 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.313703    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f1bcaac-435e-43f4-981b-9877fb5ece92-cabundle-cert\") pod \"2f1bcaac-435e-43f4-981b-9877fb5ece92\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") "
Sep 17 17:06:21.313821 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.313730    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f1bcaac-435e-43f4-981b-9877fb5ece92-kserve-provision-location\") pod \"2f1bcaac-435e-43f4-981b-9877fb5ece92\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") "
Sep 17 17:06:21.313821 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.313774    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwqv5\" (UniqueName: \"kubernetes.io/projected/2f1bcaac-435e-43f4-981b-9877fb5ece92-kube-api-access-hwqv5\") pod \"2f1bcaac-435e-43f4-981b-9877fb5ece92\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") "
Sep 17 17:06:21.313821 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.313817    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1bcaac-435e-43f4-981b-9877fb5ece92-proxy-tls\") pod \"2f1bcaac-435e-43f4-981b-9877fb5ece92\" (UID: \"2f1bcaac-435e-43f4-981b-9877fb5ece92\") "
Sep 17 17:06:21.314016 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.313948    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1bcaac-435e-43f4-981b-9877fb5ece92-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2f1bcaac-435e-43f4-981b-9877fb5ece92" (UID: "2f1bcaac-435e-43f4-981b-9877fb5ece92"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:06:21.314063 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.314028    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1bcaac-435e-43f4-981b-9877fb5ece92-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2f1bcaac-435e-43f4-981b-9877fb5ece92" (UID: "2f1bcaac-435e-43f4-981b-9877fb5ece92"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:06:21.314103 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.314060    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f1bcaac-435e-43f4-981b-9877fb5ece92-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:06:21.315818 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.315792    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1bcaac-435e-43f4-981b-9877fb5ece92-kube-api-access-hwqv5" (OuterVolumeSpecName: "kube-api-access-hwqv5") pod "2f1bcaac-435e-43f4-981b-9877fb5ece92" (UID: "2f1bcaac-435e-43f4-981b-9877fb5ece92"). InnerVolumeSpecName "kube-api-access-hwqv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:06:21.315960 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.315874    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1bcaac-435e-43f4-981b-9877fb5ece92-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f1bcaac-435e-43f4-981b-9877fb5ece92" (UID: "2f1bcaac-435e-43f4-981b-9877fb5ece92"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:06:21.318216 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.318189    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1bcaac-435e-43f4-981b-9877fb5ece92-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f1bcaac-435e-43f4-981b-9877fb5ece92" (UID: "2f1bcaac-435e-43f4-981b-9877fb5ece92"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:06:21.415142 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.415121    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwqv5\" (UniqueName: \"kubernetes.io/projected/2f1bcaac-435e-43f4-981b-9877fb5ece92-kube-api-access-hwqv5\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:06:21.415142 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.415140    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1bcaac-435e-43f4-981b-9877fb5ece92-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:06:21.415282 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.415149    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f1bcaac-435e-43f4-981b-9877fb5ece92-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:06:21.415282 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:21.415174    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f1bcaac-435e-43f4-981b-9877fb5ece92-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:06:22.066422 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:22.066393    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"
Sep 17 17:06:22.066876 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:22.066393    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m" event={"ID":"2f1bcaac-435e-43f4-981b-9877fb5ece92","Type":"ContainerDied","Data":"f5c2ff3461804aef04cecee1bfcd165829038525cb33e1b243bf1fac1a52be3f"}
Sep 17 17:06:22.066876 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:22.066540    2577 scope.go:117] "RemoveContainer" containerID="0889a1ed5aaa3789d107b029524ac6adf3425fe6c99f06efeee9c31ae0a6d9d2"
Sep 17 17:06:22.075262 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:22.075241    2577 scope.go:117] "RemoveContainer" containerID="923b5858d8ae9bd390b110b5d292232e767a61477c150cd503fd3f193ff28414"
Sep 17 17:06:22.084674 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:22.084652    2577 scope.go:117] "RemoveContainer" containerID="cac1dac6e60ea282b532e8298118a7e22d36d66c335cdd7625bbcb83c413c373"
Sep 17 17:06:22.089201 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:22.089181    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"]
Sep 17 17:06:22.092219 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:22.092190    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-nc86m"]
Sep 17 17:06:22.446875 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:22.446808    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f1bcaac-435e-43f4-981b-9877fb5ece92" path="/var/lib/kubelet/pods/2f1bcaac-435e-43f4-981b-9877fb5ece92/volumes"
Sep 17 17:06:22.498176 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:22.498142    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 17:06:22.499316 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:22.499295    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 17:06:25.011687 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.011651    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"]
Sep 17 17:06:25.012025 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.011950    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1bcaac-435e-43f4-981b-9877fb5ece92" containerName="kube-rbac-proxy"
Sep 17 17:06:25.012025 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.011961    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1bcaac-435e-43f4-981b-9877fb5ece92" containerName="kube-rbac-proxy"
Sep 17 17:06:25.012025 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.011972    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1bcaac-435e-43f4-981b-9877fb5ece92" containerName="kserve-container"
Sep 17 17:06:25.012025 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.011977    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1bcaac-435e-43f4-981b-9877fb5ece92" containerName="kserve-container"
Sep 17 17:06:25.012025 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.011988    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1bcaac-435e-43f4-981b-9877fb5ece92" containerName="storage-initializer"
Sep 17 17:06:25.012025 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.011994    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1bcaac-435e-43f4-981b-9877fb5ece92" containerName="storage-initializer"
Sep 17 17:06:25.012256 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.012043    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f1bcaac-435e-43f4-981b-9877fb5ece92" containerName="kserve-container"
Sep 17 17:06:25.012256 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.012052    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f1bcaac-435e-43f4-981b-9877fb5ece92" containerName="kube-rbac-proxy"
Sep 17 17:06:25.014978 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.014962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.018073 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.018047    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 17 17:06:25.018073 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.018067    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:06:25.018328 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.018082    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:06:25.018328 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.018193    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:06:25.018328 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.018283    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 17 17:06:25.019315 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.019294    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:06:25.019401 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.019353    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:06:25.025295 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.025271    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"]
Sep 17 17:06:25.139986 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.139960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a357abcd-1b01-403c-9a34-fe6cad9c6d63-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.140144 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.139995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a357abcd-1b01-403c-9a34-fe6cad9c6d63-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.140144 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.140025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.140144 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.140104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.140144 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.140137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b25kl\" (UniqueName: \"kubernetes.io/projected/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kube-api-access-b25kl\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.240753 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.240716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a357abcd-1b01-403c-9a34-fe6cad9c6d63-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.240866 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.240768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a357abcd-1b01-403c-9a34-fe6cad9c6d63-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.240866 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.240800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.240866 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.240832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.241027 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.240865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b25kl\" (UniqueName: \"kubernetes.io/projected/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kube-api-access-b25kl\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.241402 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.241374    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.241622 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.241600    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a357abcd-1b01-403c-9a34-fe6cad9c6d63-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.241695 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.241642    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.243314 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.243295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a357abcd-1b01-403c-9a34-fe6cad9c6d63-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.249687 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.249661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b25kl\" (UniqueName: \"kubernetes.io/projected/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kube-api-access-b25kl\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.325922 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.325891    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:25.599408 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:25.599383    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"]
Sep 17 17:06:25.601716 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:06:25.601687    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda357abcd_1b01_403c_9a34_fe6cad9c6d63.slice/crio-578b817a04beaca60677cb02befbd0d1b2e74c735e15b21820a5d0c0f754d572 WatchSource:0}: Error finding container 578b817a04beaca60677cb02befbd0d1b2e74c735e15b21820a5d0c0f754d572: Status 404 returned error can't find the container with id 578b817a04beaca60677cb02befbd0d1b2e74c735e15b21820a5d0c0f754d572
Sep 17 17:06:26.085375 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:26.085343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs" event={"ID":"a357abcd-1b01-403c-9a34-fe6cad9c6d63","Type":"ContainerStarted","Data":"83eaecf24cfee3215aba2ba9af682dc3265d277fa42ff5bb1796b939bdf3f52e"}
Sep 17 17:06:26.085825 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:26.085398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs" event={"ID":"a357abcd-1b01-403c-9a34-fe6cad9c6d63","Type":"ContainerStarted","Data":"578b817a04beaca60677cb02befbd0d1b2e74c735e15b21820a5d0c0f754d572"}
Sep 17 17:06:35.127645 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:35.127616    2577 generic.go:356] "Generic (PLEG): container finished" podID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerID="83eaecf24cfee3215aba2ba9af682dc3265d277fa42ff5bb1796b939bdf3f52e" exitCode=0
Sep 17 17:06:35.127928 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:35.127685    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs" event={"ID":"a357abcd-1b01-403c-9a34-fe6cad9c6d63","Type":"ContainerDied","Data":"83eaecf24cfee3215aba2ba9af682dc3265d277fa42ff5bb1796b939bdf3f52e"}
Sep 17 17:06:36.134520 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:36.134484    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs" event={"ID":"a357abcd-1b01-403c-9a34-fe6cad9c6d63","Type":"ContainerStarted","Data":"ae5f47d3b9dc91f7de7b74f0aeb0e62018d6670764f77675873fdfcea9120c4b"}
Sep 17 17:06:36.134897 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:36.134547    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs" event={"ID":"a357abcd-1b01-403c-9a34-fe6cad9c6d63","Type":"ContainerStarted","Data":"daadf5608d2fb8598f1928b00a425e9556256abf76f41e1fcf888df696aa58fb"}
Sep 17 17:06:36.134897 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:36.134735    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:36.153777 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:36.153730    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs" podStartSLOduration=2.691825753 podStartE2EDuration="12.153715476s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.461889723s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:06:36.152468243 +0000 UTC m=+1514.177376533" watchObservedRunningTime="2026-09-17 17:06:36.153715476 +0000 UTC m=+1514.178623763"
Sep 17 17:06:37.139240 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:37.139207    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:37.140716 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:37.140688    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs" podUID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 17:06:38.143898 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:38.143854    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs" podUID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 17:06:43.148436 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:43.148402    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:06:43.149026 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:43.148995    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs" podUID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 17:06:53.149308 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:06:53.149279    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:07:07.275340 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:07.275305    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"]
Sep 17 17:07:07.275927 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:07.275535    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs" podUID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerName="kserve-container" containerID="cri-o://daadf5608d2fb8598f1928b00a425e9556256abf76f41e1fcf888df696aa58fb" gracePeriod=30
Sep 17 17:07:07.275927 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:07.275558    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs" podUID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerName="kube-rbac-proxy" containerID="cri-o://ae5f47d3b9dc91f7de7b74f0aeb0e62018d6670764f77675873fdfcea9120c4b" gracePeriod=30
Sep 17 17:07:08.144130 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:08.144080    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs" podUID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 17 17:07:08.278180 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:08.278133    2577 generic.go:356] "Generic (PLEG): container finished" podID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerID="ae5f47d3b9dc91f7de7b74f0aeb0e62018d6670764f77675873fdfcea9120c4b" exitCode=2
Sep 17 17:07:08.278558 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:08.278208    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs" event={"ID":"a357abcd-1b01-403c-9a34-fe6cad9c6d63","Type":"ContainerDied","Data":"ae5f47d3b9dc91f7de7b74f0aeb0e62018d6670764f77675873fdfcea9120c4b"}
Sep 17 17:07:08.919925 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:08.919901    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:07:08.974925 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:08.974860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b25kl\" (UniqueName: \"kubernetes.io/projected/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kube-api-access-b25kl\") pod \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") "
Sep 17 17:07:08.974925 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:08.974911    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kserve-provision-location\") pod \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") "
Sep 17 17:07:08.975092 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:08.974934    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a357abcd-1b01-403c-9a34-fe6cad9c6d63-cabundle-cert\") pod \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") "
Sep 17 17:07:08.975092 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:08.974974    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kube-rbac-proxy-sar-config\") pod \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") "
Sep 17 17:07:08.975092 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:08.975002    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a357abcd-1b01-403c-9a34-fe6cad9c6d63-proxy-tls\") pod \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\" (UID: \"a357abcd-1b01-403c-9a34-fe6cad9c6d63\") "
Sep 17 17:07:08.975339 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:08.975311    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a357abcd-1b01-403c-9a34-fe6cad9c6d63-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a357abcd-1b01-403c-9a34-fe6cad9c6d63" (UID: "a357abcd-1b01-403c-9a34-fe6cad9c6d63"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:07:08.975449 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:08.975340    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a357abcd-1b01-403c-9a34-fe6cad9c6d63" (UID: "a357abcd-1b01-403c-9a34-fe6cad9c6d63"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:07:08.975903 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:08.975876    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a357abcd-1b01-403c-9a34-fe6cad9c6d63" (UID: "a357abcd-1b01-403c-9a34-fe6cad9c6d63"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:07:08.976954 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:08.976916    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a357abcd-1b01-403c-9a34-fe6cad9c6d63-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a357abcd-1b01-403c-9a34-fe6cad9c6d63" (UID: "a357abcd-1b01-403c-9a34-fe6cad9c6d63"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:07:08.977038 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:08.977010    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kube-api-access-b25kl" (OuterVolumeSpecName: "kube-api-access-b25kl") pod "a357abcd-1b01-403c-9a34-fe6cad9c6d63" (UID: "a357abcd-1b01-403c-9a34-fe6cad9c6d63"). InnerVolumeSpecName "kube-api-access-b25kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:07:09.076065 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.076038    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b25kl\" (UniqueName: \"kubernetes.io/projected/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kube-api-access-b25kl\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:07:09.076065 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.076062    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:07:09.076253 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.076074    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a357abcd-1b01-403c-9a34-fe6cad9c6d63-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:07:09.076253 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.076082    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a357abcd-1b01-403c-9a34-fe6cad9c6d63-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:07:09.076253 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.076091    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a357abcd-1b01-403c-9a34-fe6cad9c6d63-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:07:09.284883 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.284798    2577 generic.go:356] "Generic (PLEG): container finished" podID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerID="daadf5608d2fb8598f1928b00a425e9556256abf76f41e1fcf888df696aa58fb" exitCode=0
Sep 17 17:07:09.284883 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.284852    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"
Sep 17 17:07:09.284883 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.284858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs" event={"ID":"a357abcd-1b01-403c-9a34-fe6cad9c6d63","Type":"ContainerDied","Data":"daadf5608d2fb8598f1928b00a425e9556256abf76f41e1fcf888df696aa58fb"}
Sep 17 17:07:09.285343 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.284908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs" event={"ID":"a357abcd-1b01-403c-9a34-fe6cad9c6d63","Type":"ContainerDied","Data":"578b817a04beaca60677cb02befbd0d1b2e74c735e15b21820a5d0c0f754d572"}
Sep 17 17:07:09.285343 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.284936    2577 scope.go:117] "RemoveContainer" containerID="ae5f47d3b9dc91f7de7b74f0aeb0e62018d6670764f77675873fdfcea9120c4b"
Sep 17 17:07:09.293466 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.293448    2577 scope.go:117] "RemoveContainer" containerID="daadf5608d2fb8598f1928b00a425e9556256abf76f41e1fcf888df696aa58fb"
Sep 17 17:07:09.300482 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.300464    2577 scope.go:117] "RemoveContainer" containerID="83eaecf24cfee3215aba2ba9af682dc3265d277fa42ff5bb1796b939bdf3f52e"
Sep 17 17:07:09.306820 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.306798    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"]
Sep 17 17:07:09.307352 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.307338    2577 scope.go:117] "RemoveContainer" containerID="ae5f47d3b9dc91f7de7b74f0aeb0e62018d6670764f77675873fdfcea9120c4b"
Sep 17 17:07:09.307596 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:07:09.307577    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5f47d3b9dc91f7de7b74f0aeb0e62018d6670764f77675873fdfcea9120c4b\": container with ID starting with ae5f47d3b9dc91f7de7b74f0aeb0e62018d6670764f77675873fdfcea9120c4b not found: ID does not exist" containerID="ae5f47d3b9dc91f7de7b74f0aeb0e62018d6670764f77675873fdfcea9120c4b"
Sep 17 17:07:09.307657 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.307605    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5f47d3b9dc91f7de7b74f0aeb0e62018d6670764f77675873fdfcea9120c4b"} err="failed to get container status \"ae5f47d3b9dc91f7de7b74f0aeb0e62018d6670764f77675873fdfcea9120c4b\": rpc error: code = NotFound desc = could not find container \"ae5f47d3b9dc91f7de7b74f0aeb0e62018d6670764f77675873fdfcea9120c4b\": container with ID starting with ae5f47d3b9dc91f7de7b74f0aeb0e62018d6670764f77675873fdfcea9120c4b not found: ID does not exist"
Sep 17 17:07:09.307657 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.307622    2577 scope.go:117] "RemoveContainer" containerID="daadf5608d2fb8598f1928b00a425e9556256abf76f41e1fcf888df696aa58fb"
Sep 17 17:07:09.307859 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:07:09.307844    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daadf5608d2fb8598f1928b00a425e9556256abf76f41e1fcf888df696aa58fb\": container with ID starting with daadf5608d2fb8598f1928b00a425e9556256abf76f41e1fcf888df696aa58fb not found: ID does not exist" containerID="daadf5608d2fb8598f1928b00a425e9556256abf76f41e1fcf888df696aa58fb"
Sep 17 17:07:09.307899 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.307865    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daadf5608d2fb8598f1928b00a425e9556256abf76f41e1fcf888df696aa58fb"} err="failed to get container status \"daadf5608d2fb8598f1928b00a425e9556256abf76f41e1fcf888df696aa58fb\": rpc error: code = NotFound desc = could not find container \"daadf5608d2fb8598f1928b00a425e9556256abf76f41e1fcf888df696aa58fb\": container with ID starting with daadf5608d2fb8598f1928b00a425e9556256abf76f41e1fcf888df696aa58fb not found: ID does not exist"
Sep 17 17:07:09.307899 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.307878    2577 scope.go:117] "RemoveContainer" containerID="83eaecf24cfee3215aba2ba9af682dc3265d277fa42ff5bb1796b939bdf3f52e"
Sep 17 17:07:09.308098 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:07:09.308080    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83eaecf24cfee3215aba2ba9af682dc3265d277fa42ff5bb1796b939bdf3f52e\": container with ID starting with 83eaecf24cfee3215aba2ba9af682dc3265d277fa42ff5bb1796b939bdf3f52e not found: ID does not exist" containerID="83eaecf24cfee3215aba2ba9af682dc3265d277fa42ff5bb1796b939bdf3f52e"
Sep 17 17:07:09.308146 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.308100    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83eaecf24cfee3215aba2ba9af682dc3265d277fa42ff5bb1796b939bdf3f52e"} err="failed to get container status \"83eaecf24cfee3215aba2ba9af682dc3265d277fa42ff5bb1796b939bdf3f52e\": rpc error: code = NotFound desc = could not find container \"83eaecf24cfee3215aba2ba9af682dc3265d277fa42ff5bb1796b939bdf3f52e\": container with ID starting with 83eaecf24cfee3215aba2ba9af682dc3265d277fa42ff5bb1796b939bdf3f52e not found: ID does not exist"
Sep 17 17:07:09.312657 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:09.312636    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-wlnrs"]
Sep 17 17:07:10.448743 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:10.448711    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" path="/var/lib/kubelet/pods/a357abcd-1b01-403c-9a34-fe6cad9c6d63/volumes"
Sep 17 17:07:12.505585 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.505553    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"]
Sep 17 17:07:12.505928 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.505833    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerName="kserve-container"
Sep 17 17:07:12.505928 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.505844    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerName="kserve-container"
Sep 17 17:07:12.505928 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.505864    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerName="kube-rbac-proxy"
Sep 17 17:07:12.505928 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.505870    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerName="kube-rbac-proxy"
Sep 17 17:07:12.505928 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.505877    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerName="storage-initializer"
Sep 17 17:07:12.505928 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.505883    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerName="storage-initializer"
Sep 17 17:07:12.505928 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.505929    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerName="kserve-container"
Sep 17 17:07:12.506144 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.505939    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a357abcd-1b01-403c-9a34-fe6cad9c6d63" containerName="kube-rbac-proxy"
Sep 17 17:07:12.510608 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.510589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.513286 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.513262    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 17 17:07:12.513645 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.513625    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:07:12.513645 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.513635    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:07:12.513645 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.513662    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:07:12.513881 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.513745    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 17 17:07:12.514506 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.514487    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:07:12.514787 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.514774    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:07:12.519143 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.519123    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"]
Sep 17 17:07:12.597593 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.597568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.597698 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.597615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.597698 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.597637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t94r4\" (UniqueName: \"kubernetes.io/projected/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kube-api-access-t94r4\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.597698 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.597658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.597698 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.597677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.698740 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.698717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.698860 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.698749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t94r4\" (UniqueName: \"kubernetes.io/projected/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kube-api-access-t94r4\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.698860 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.698770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.698860 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.698798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.698860 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.698838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.699122 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.699102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.699544 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.699515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.699648 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.699548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.701301 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.701280    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.707503 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.707482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t94r4\" (UniqueName: \"kubernetes.io/projected/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kube-api-access-t94r4\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:12.822245 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:12.822225    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:13.095226 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:13.095201    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"]
Sep 17 17:07:13.097292 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:07:13.097254    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c944ccb_c4a5_42f4_9d56_0cc140b869ff.slice/crio-24cf31aba96c304df2bf4260965eb06618f7d650884d5b54791394207da8ba60 WatchSource:0}: Error finding container 24cf31aba96c304df2bf4260965eb06618f7d650884d5b54791394207da8ba60: Status 404 returned error can't find the container with id 24cf31aba96c304df2bf4260965eb06618f7d650884d5b54791394207da8ba60
Sep 17 17:07:13.305343 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:13.305306    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb" event={"ID":"9c944ccb-c4a5-42f4-9d56-0cc140b869ff","Type":"ContainerStarted","Data":"18d76578f86a472647dde506b1f114514bd6834e2d04b651e3ae61fa170cd515"}
Sep 17 17:07:13.305500 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:13.305375    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb" event={"ID":"9c944ccb-c4a5-42f4-9d56-0cc140b869ff","Type":"ContainerStarted","Data":"24cf31aba96c304df2bf4260965eb06618f7d650884d5b54791394207da8ba60"}
Sep 17 17:07:18.329751 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:18.329720    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerID="18d76578f86a472647dde506b1f114514bd6834e2d04b651e3ae61fa170cd515" exitCode=0
Sep 17 17:07:18.330179 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:18.329793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb" event={"ID":"9c944ccb-c4a5-42f4-9d56-0cc140b869ff","Type":"ContainerDied","Data":"18d76578f86a472647dde506b1f114514bd6834e2d04b651e3ae61fa170cd515"}
Sep 17 17:07:19.337002 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:19.336969    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb" event={"ID":"9c944ccb-c4a5-42f4-9d56-0cc140b869ff","Type":"ContainerStarted","Data":"9c0714bb0fe95a330bc485afa434d8680b6154616e5b99e731cbb22c0d311f20"}
Sep 17 17:07:19.337415 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:19.337029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb" event={"ID":"9c944ccb-c4a5-42f4-9d56-0cc140b869ff","Type":"ContainerStarted","Data":"b5129fd97d0a5ef8b8f4c13cf99c80b6596199d1863039377327dbfea72c884c"}
Sep 17 17:07:19.337415 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:19.337291    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:19.337505 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:19.337440    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:19.338990 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:19.338964    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb" podUID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 17:07:19.356106 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:19.356069    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb" podStartSLOduration=2.796089695 podStartE2EDuration="7.356056159s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.559966464s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:07:19.354387927 +0000 UTC m=+1557.379296216" watchObservedRunningTime="2026-09-17 17:07:19.356056159 +0000 UTC m=+1557.380964448"
Sep 17 17:07:20.341620 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:20.341578    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb" podUID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 17:07:25.346754 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:25.346725    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:25.347217 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:25.347191    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb" podUID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 17:07:35.348542 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:35.348509    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:47.759374 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:47.759326    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"]
Sep 17 17:07:47.759810 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:47.759618    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb" podUID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerName="kserve-container" containerID="cri-o://b5129fd97d0a5ef8b8f4c13cf99c80b6596199d1863039377327dbfea72c884c" gracePeriod=30
Sep 17 17:07:47.759810 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:47.759646    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb" podUID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerName="kube-rbac-proxy" containerID="cri-o://9c0714bb0fe95a330bc485afa434d8680b6154616e5b99e731cbb22c0d311f20" gracePeriod=30
Sep 17 17:07:48.467329 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:48.467294    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerID="9c0714bb0fe95a330bc485afa434d8680b6154616e5b99e731cbb22c0d311f20" exitCode=2
Sep 17 17:07:48.467541 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:48.467373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb" event={"ID":"9c944ccb-c4a5-42f4-9d56-0cc140b869ff","Type":"ContainerDied","Data":"9c0714bb0fe95a330bc485afa434d8680b6154616e5b99e731cbb22c0d311f20"}
Sep 17 17:07:49.006435 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.006414    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:49.176271 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.176172    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t94r4\" (UniqueName: \"kubernetes.io/projected/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kube-api-access-t94r4\") pod \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") "
Sep 17 17:07:49.176271 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.176253    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-cabundle-cert\") pod \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") "
Sep 17 17:07:49.176701 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.176301    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-proxy-tls\") pod \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") "
Sep 17 17:07:49.176701 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.176354    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kube-rbac-proxy-sar-config\") pod \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") "
Sep 17 17:07:49.176701 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.176384    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kserve-provision-location\") pod \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\" (UID: \"9c944ccb-c4a5-42f4-9d56-0cc140b869ff\") "
Sep 17 17:07:49.176701 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.176673    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9c944ccb-c4a5-42f4-9d56-0cc140b869ff" (UID: "9c944ccb-c4a5-42f4-9d56-0cc140b869ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:07:49.176847 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.176716    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9c944ccb-c4a5-42f4-9d56-0cc140b869ff" (UID: "9c944ccb-c4a5-42f4-9d56-0cc140b869ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:07:49.177296 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.177273    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c944ccb-c4a5-42f4-9d56-0cc140b869ff" (UID: "9c944ccb-c4a5-42f4-9d56-0cc140b869ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:07:49.178312 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.178287    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kube-api-access-t94r4" (OuterVolumeSpecName: "kube-api-access-t94r4") pod "9c944ccb-c4a5-42f4-9d56-0cc140b869ff" (UID: "9c944ccb-c4a5-42f4-9d56-0cc140b869ff"). InnerVolumeSpecName "kube-api-access-t94r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:07:49.178668 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.178643    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9c944ccb-c4a5-42f4-9d56-0cc140b869ff" (UID: "9c944ccb-c4a5-42f4-9d56-0cc140b869ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:07:49.277090 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.277059    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:07:49.277090 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.277085    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:07:49.277090 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.277096    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:07:49.277319 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.277124    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t94r4\" (UniqueName: \"kubernetes.io/projected/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-kube-api-access-t94r4\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:07:49.277319 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.277135    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c944ccb-c4a5-42f4-9d56-0cc140b869ff-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:07:49.473827 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.473755    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerID="b5129fd97d0a5ef8b8f4c13cf99c80b6596199d1863039377327dbfea72c884c" exitCode=0
Sep 17 17:07:49.473827 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.473796    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb" event={"ID":"9c944ccb-c4a5-42f4-9d56-0cc140b869ff","Type":"ContainerDied","Data":"b5129fd97d0a5ef8b8f4c13cf99c80b6596199d1863039377327dbfea72c884c"}
Sep 17 17:07:49.473827 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.473822    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"
Sep 17 17:07:49.474016 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.473853    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb" event={"ID":"9c944ccb-c4a5-42f4-9d56-0cc140b869ff","Type":"ContainerDied","Data":"24cf31aba96c304df2bf4260965eb06618f7d650884d5b54791394207da8ba60"}
Sep 17 17:07:49.474016 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.473879    2577 scope.go:117] "RemoveContainer" containerID="9c0714bb0fe95a330bc485afa434d8680b6154616e5b99e731cbb22c0d311f20"
Sep 17 17:07:49.483005 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.482990    2577 scope.go:117] "RemoveContainer" containerID="b5129fd97d0a5ef8b8f4c13cf99c80b6596199d1863039377327dbfea72c884c"
Sep 17 17:07:49.492129 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.492112    2577 scope.go:117] "RemoveContainer" containerID="18d76578f86a472647dde506b1f114514bd6834e2d04b651e3ae61fa170cd515"
Sep 17 17:07:49.495680 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.495656    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"]
Sep 17 17:07:49.499351 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.499328    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rn2qb"]
Sep 17 17:07:49.500105 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.500090    2577 scope.go:117] "RemoveContainer" containerID="9c0714bb0fe95a330bc485afa434d8680b6154616e5b99e731cbb22c0d311f20"
Sep 17 17:07:49.500385 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:07:49.500367    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c0714bb0fe95a330bc485afa434d8680b6154616e5b99e731cbb22c0d311f20\": container with ID starting with 9c0714bb0fe95a330bc485afa434d8680b6154616e5b99e731cbb22c0d311f20 not found: ID does not exist" containerID="9c0714bb0fe95a330bc485afa434d8680b6154616e5b99e731cbb22c0d311f20"
Sep 17 17:07:49.500449 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.500393    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c0714bb0fe95a330bc485afa434d8680b6154616e5b99e731cbb22c0d311f20"} err="failed to get container status \"9c0714bb0fe95a330bc485afa434d8680b6154616e5b99e731cbb22c0d311f20\": rpc error: code = NotFound desc = could not find container \"9c0714bb0fe95a330bc485afa434d8680b6154616e5b99e731cbb22c0d311f20\": container with ID starting with 9c0714bb0fe95a330bc485afa434d8680b6154616e5b99e731cbb22c0d311f20 not found: ID does not exist"
Sep 17 17:07:49.500449 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.500409    2577 scope.go:117] "RemoveContainer" containerID="b5129fd97d0a5ef8b8f4c13cf99c80b6596199d1863039377327dbfea72c884c"
Sep 17 17:07:49.500682 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:07:49.500665    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5129fd97d0a5ef8b8f4c13cf99c80b6596199d1863039377327dbfea72c884c\": container with ID starting with b5129fd97d0a5ef8b8f4c13cf99c80b6596199d1863039377327dbfea72c884c not found: ID does not exist" containerID="b5129fd97d0a5ef8b8f4c13cf99c80b6596199d1863039377327dbfea72c884c"
Sep 17 17:07:49.500736 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.500687    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5129fd97d0a5ef8b8f4c13cf99c80b6596199d1863039377327dbfea72c884c"} err="failed to get container status \"b5129fd97d0a5ef8b8f4c13cf99c80b6596199d1863039377327dbfea72c884c\": rpc error: code = NotFound desc = could not find container \"b5129fd97d0a5ef8b8f4c13cf99c80b6596199d1863039377327dbfea72c884c\": container with ID starting with b5129fd97d0a5ef8b8f4c13cf99c80b6596199d1863039377327dbfea72c884c not found: ID does not exist"
Sep 17 17:07:49.500736 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.500701    2577 scope.go:117] "RemoveContainer" containerID="18d76578f86a472647dde506b1f114514bd6834e2d04b651e3ae61fa170cd515"
Sep 17 17:07:49.500906 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:07:49.500890    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d76578f86a472647dde506b1f114514bd6834e2d04b651e3ae61fa170cd515\": container with ID starting with 18d76578f86a472647dde506b1f114514bd6834e2d04b651e3ae61fa170cd515 not found: ID does not exist" containerID="18d76578f86a472647dde506b1f114514bd6834e2d04b651e3ae61fa170cd515"
Sep 17 17:07:49.500947 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:49.500908    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d76578f86a472647dde506b1f114514bd6834e2d04b651e3ae61fa170cd515"} err="failed to get container status \"18d76578f86a472647dde506b1f114514bd6834e2d04b651e3ae61fa170cd515\": rpc error: code = NotFound desc = could not find container \"18d76578f86a472647dde506b1f114514bd6834e2d04b651e3ae61fa170cd515\": container with ID starting with 18d76578f86a472647dde506b1f114514bd6834e2d04b651e3ae61fa170cd515 not found: ID does not exist"
Sep 17 17:07:50.447903 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:50.447869    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" path="/var/lib/kubelet/pods/9c944ccb-c4a5-42f4-9d56-0cc140b869ff/volumes"
Sep 17 17:07:53.009551 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.009522    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"]
Sep 17 17:07:53.009911 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.009817    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerName="kube-rbac-proxy"
Sep 17 17:07:53.009911 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.009828    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerName="kube-rbac-proxy"
Sep 17 17:07:53.009911 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.009836    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerName="storage-initializer"
Sep 17 17:07:53.009911 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.009842    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerName="storage-initializer"
Sep 17 17:07:53.009911 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.009848    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerName="kserve-container"
Sep 17 17:07:53.009911 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.009854    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerName="kserve-container"
Sep 17 17:07:53.009911 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.009913    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerName="kserve-container"
Sep 17 17:07:53.010138 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.009919    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c944ccb-c4a5-42f4-9d56-0cc140b869ff" containerName="kube-rbac-proxy"
Sep 17 17:07:53.014500 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.014482    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.017136 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.017109    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:07:53.017136 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.017129    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:07:53.017136 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.017110    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:07:53.017397 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.017110    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:07:53.018184 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.018154    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 17:07:53.018271 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.018199    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:07:53.018312 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.018264    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 17:07:53.023605 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.023584    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"]
Sep 17 17:07:53.107598 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.107572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75ede61f-d515-4b5d-bc57-ecbf5188c163-cabundle-cert\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.107717 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.107620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75ede61f-d515-4b5d-bc57-ecbf5188c163-kserve-provision-location\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.107717 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.107666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75ede61f-d515-4b5d-bc57-ecbf5188c163-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.107717 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.107687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75ede61f-d515-4b5d-bc57-ecbf5188c163-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.107717 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.107703    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwvgw\" (UniqueName: \"kubernetes.io/projected/75ede61f-d515-4b5d-bc57-ecbf5188c163-kube-api-access-jwvgw\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.208884 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.208858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75ede61f-d515-4b5d-bc57-ecbf5188c163-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.208971 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.208888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75ede61f-d515-4b5d-bc57-ecbf5188c163-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.208971 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.208906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwvgw\" (UniqueName: \"kubernetes.io/projected/75ede61f-d515-4b5d-bc57-ecbf5188c163-kube-api-access-jwvgw\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.208971 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.208926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75ede61f-d515-4b5d-bc57-ecbf5188c163-cabundle-cert\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.209085 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.208978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75ede61f-d515-4b5d-bc57-ecbf5188c163-kserve-provision-location\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.209085 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:07:53.209029    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 17:07:53.209207 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:07:53.209085    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75ede61f-d515-4b5d-bc57-ecbf5188c163-proxy-tls podName:75ede61f-d515-4b5d-bc57-ecbf5188c163 nodeName:}" failed. No retries permitted until 2026-09-17 17:07:53.709069341 +0000 UTC m=+1591.733977607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/75ede61f-d515-4b5d-bc57-ecbf5188c163-proxy-tls") pod "isvc-canary-create-predictor-8cb968449-4j5zz" (UID: "75ede61f-d515-4b5d-bc57-ecbf5188c163") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 17:07:53.209398 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.209381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75ede61f-d515-4b5d-bc57-ecbf5188c163-kserve-provision-location\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.209701 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.209680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75ede61f-d515-4b5d-bc57-ecbf5188c163-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.209740 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.209680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75ede61f-d515-4b5d-bc57-ecbf5188c163-cabundle-cert\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.217674 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.217656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwvgw\" (UniqueName: \"kubernetes.io/projected/75ede61f-d515-4b5d-bc57-ecbf5188c163-kube-api-access-jwvgw\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.712577 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.712549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75ede61f-d515-4b5d-bc57-ecbf5188c163-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.714815 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.714790    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75ede61f-d515-4b5d-bc57-ecbf5188c163-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-4j5zz\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:53.924594 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:53.924551    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:07:54.194735 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:54.194701    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"]
Sep 17 17:07:54.197066 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:07:54.197039    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ede61f_d515_4b5d_bc57_ecbf5188c163.slice/crio-0271b15e6c8b072b05030e93a56a7d10bc6e3557eab8f6df30a9839cf103810e WatchSource:0}: Error finding container 0271b15e6c8b072b05030e93a56a7d10bc6e3557eab8f6df30a9839cf103810e: Status 404 returned error can't find the container with id 0271b15e6c8b072b05030e93a56a7d10bc6e3557eab8f6df30a9839cf103810e
Sep 17 17:07:54.497650 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:54.497572    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" event={"ID":"75ede61f-d515-4b5d-bc57-ecbf5188c163","Type":"ContainerStarted","Data":"af90734ff049a8a7f18d3dffa4dababddbe4d0773da341cbee54041cf4bc49aa"}
Sep 17 17:07:54.497650 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:54.497637    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" event={"ID":"75ede61f-d515-4b5d-bc57-ecbf5188c163","Type":"ContainerStarted","Data":"0271b15e6c8b072b05030e93a56a7d10bc6e3557eab8f6df30a9839cf103810e"}
Sep 17 17:07:58.516307 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:58.516224    2577 generic.go:356] "Generic (PLEG): container finished" podID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerID="af90734ff049a8a7f18d3dffa4dababddbe4d0773da341cbee54041cf4bc49aa" exitCode=0
Sep 17 17:07:58.516307 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:58.516291    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" event={"ID":"75ede61f-d515-4b5d-bc57-ecbf5188c163","Type":"ContainerDied","Data":"af90734ff049a8a7f18d3dffa4dababddbe4d0773da341cbee54041cf4bc49aa"}
Sep 17 17:07:58.517354 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:07:58.517338    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:08:12.593823 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:08:12.593791    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" event={"ID":"75ede61f-d515-4b5d-bc57-ecbf5188c163","Type":"ContainerStarted","Data":"dcdfff067051e7d139163c5e5e44af45ea2bcb0daed612ec8f3298582c35fe46"}
Sep 17 17:08:12.594241 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:08:12.593850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" event={"ID":"75ede61f-d515-4b5d-bc57-ecbf5188c163","Type":"ContainerStarted","Data":"f8e10e726e23d06ddfe502a7d0960897b1053e2a67bb943608906fb7319befc3"}
Sep 17 17:08:12.594241 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:08:12.594028    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:08:12.612454 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:08:12.612411    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" podStartSLOduration=3.210028478 podStartE2EDuration="20.612386369s" totalImagesPullingTime="13.529609889s" totalInitContainerRuntime="3.872748002s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:08:12.611515469 +0000 UTC m=+1610.636423967" watchObservedRunningTime="2026-09-17 17:08:12.612386369 +0000 UTC m=+1610.637294658"
Sep 17 17:08:13.598540 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:08:13.598506    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:08:13.599546 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:08:13.599516    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:08:14.603090 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:08:14.603048    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:08:19.607313 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:08:19.607285    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:08:19.607834 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:08:19.607805    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:08:29.608107 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:08:29.608072    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:08:39.607867 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:08:39.607823    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:08:49.608364 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:08:49.608325    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:08:59.608359 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:08:59.608313    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:09:09.608059 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:09.608014    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:09:19.607788 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:19.607740    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:09:29.608866 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:29.608836    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:09:32.990508 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:32.990462    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"]
Sep 17 17:09:32.990919 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:32.990686    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kserve-container" containerID="cri-o://f8e10e726e23d06ddfe502a7d0960897b1053e2a67bb943608906fb7319befc3" gracePeriod=30
Sep 17 17:09:32.990919 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:32.990725    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kube-rbac-proxy" containerID="cri-o://dcdfff067051e7d139163c5e5e44af45ea2bcb0daed612ec8f3298582c35fe46" gracePeriod=30
Sep 17 17:09:33.960337 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:33.960306    2577 generic.go:356] "Generic (PLEG): container finished" podID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerID="dcdfff067051e7d139163c5e5e44af45ea2bcb0daed612ec8f3298582c35fe46" exitCode=2
Sep 17 17:09:33.960505 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:33.960379    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" event={"ID":"75ede61f-d515-4b5d-bc57-ecbf5188c163","Type":"ContainerDied","Data":"dcdfff067051e7d139163c5e5e44af45ea2bcb0daed612ec8f3298582c35fe46"}
Sep 17 17:09:34.603772 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:34.603729    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 17 17:09:36.933282 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.933256    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:09:36.949217 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.949190    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75ede61f-d515-4b5d-bc57-ecbf5188c163-proxy-tls\") pod \"75ede61f-d515-4b5d-bc57-ecbf5188c163\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") "
Sep 17 17:09:36.949335 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.949234    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwvgw\" (UniqueName: \"kubernetes.io/projected/75ede61f-d515-4b5d-bc57-ecbf5188c163-kube-api-access-jwvgw\") pod \"75ede61f-d515-4b5d-bc57-ecbf5188c163\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") "
Sep 17 17:09:36.949335 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.949273    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75ede61f-d515-4b5d-bc57-ecbf5188c163-cabundle-cert\") pod \"75ede61f-d515-4b5d-bc57-ecbf5188c163\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") "
Sep 17 17:09:36.949335 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.949329    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75ede61f-d515-4b5d-bc57-ecbf5188c163-kube-rbac-proxy-sar-config\") pod \"75ede61f-d515-4b5d-bc57-ecbf5188c163\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") "
Sep 17 17:09:36.949498 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.949368    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75ede61f-d515-4b5d-bc57-ecbf5188c163-kserve-provision-location\") pod \"75ede61f-d515-4b5d-bc57-ecbf5188c163\" (UID: \"75ede61f-d515-4b5d-bc57-ecbf5188c163\") "
Sep 17 17:09:36.949709 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.949679    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ede61f-d515-4b5d-bc57-ecbf5188c163-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "75ede61f-d515-4b5d-bc57-ecbf5188c163" (UID: "75ede61f-d515-4b5d-bc57-ecbf5188c163"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:09:36.949709 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.949626    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ede61f-d515-4b5d-bc57-ecbf5188c163-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75ede61f-d515-4b5d-bc57-ecbf5188c163" (UID: "75ede61f-d515-4b5d-bc57-ecbf5188c163"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:09:36.949863 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.949736    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ede61f-d515-4b5d-bc57-ecbf5188c163-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75ede61f-d515-4b5d-bc57-ecbf5188c163" (UID: "75ede61f-d515-4b5d-bc57-ecbf5188c163"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:36.951401 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.951380    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ede61f-d515-4b5d-bc57-ecbf5188c163-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75ede61f-d515-4b5d-bc57-ecbf5188c163" (UID: "75ede61f-d515-4b5d-bc57-ecbf5188c163"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:09:36.951401 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.951396    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ede61f-d515-4b5d-bc57-ecbf5188c163-kube-api-access-jwvgw" (OuterVolumeSpecName: "kube-api-access-jwvgw") pod "75ede61f-d515-4b5d-bc57-ecbf5188c163" (UID: "75ede61f-d515-4b5d-bc57-ecbf5188c163"). InnerVolumeSpecName "kube-api-access-jwvgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:09:36.974351 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.974304    2577 generic.go:356] "Generic (PLEG): container finished" podID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerID="f8e10e726e23d06ddfe502a7d0960897b1053e2a67bb943608906fb7319befc3" exitCode=0
Sep 17 17:09:36.974438 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.974356    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" event={"ID":"75ede61f-d515-4b5d-bc57-ecbf5188c163","Type":"ContainerDied","Data":"f8e10e726e23d06ddfe502a7d0960897b1053e2a67bb943608906fb7319befc3"}
Sep 17 17:09:36.974438 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.974364    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"
Sep 17 17:09:36.974438 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.974401    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz" event={"ID":"75ede61f-d515-4b5d-bc57-ecbf5188c163","Type":"ContainerDied","Data":"0271b15e6c8b072b05030e93a56a7d10bc6e3557eab8f6df30a9839cf103810e"}
Sep 17 17:09:36.974438 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.974426    2577 scope.go:117] "RemoveContainer" containerID="dcdfff067051e7d139163c5e5e44af45ea2bcb0daed612ec8f3298582c35fe46"
Sep 17 17:09:36.984612 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.984592    2577 scope.go:117] "RemoveContainer" containerID="f8e10e726e23d06ddfe502a7d0960897b1053e2a67bb943608906fb7319befc3"
Sep 17 17:09:36.991613 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.991597    2577 scope.go:117] "RemoveContainer" containerID="af90734ff049a8a7f18d3dffa4dababddbe4d0773da341cbee54041cf4bc49aa"
Sep 17 17:09:36.997822 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.997804    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"]
Sep 17 17:09:36.998559 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.998546    2577 scope.go:117] "RemoveContainer" containerID="dcdfff067051e7d139163c5e5e44af45ea2bcb0daed612ec8f3298582c35fe46"
Sep 17 17:09:36.998822 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:09:36.998803    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcdfff067051e7d139163c5e5e44af45ea2bcb0daed612ec8f3298582c35fe46\": container with ID starting with dcdfff067051e7d139163c5e5e44af45ea2bcb0daed612ec8f3298582c35fe46 not found: ID does not exist" containerID="dcdfff067051e7d139163c5e5e44af45ea2bcb0daed612ec8f3298582c35fe46"
Sep 17 17:09:36.998897 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.998832    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcdfff067051e7d139163c5e5e44af45ea2bcb0daed612ec8f3298582c35fe46"} err="failed to get container status \"dcdfff067051e7d139163c5e5e44af45ea2bcb0daed612ec8f3298582c35fe46\": rpc error: code = NotFound desc = could not find container \"dcdfff067051e7d139163c5e5e44af45ea2bcb0daed612ec8f3298582c35fe46\": container with ID starting with dcdfff067051e7d139163c5e5e44af45ea2bcb0daed612ec8f3298582c35fe46 not found: ID does not exist"
Sep 17 17:09:36.998897 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.998856    2577 scope.go:117] "RemoveContainer" containerID="f8e10e726e23d06ddfe502a7d0960897b1053e2a67bb943608906fb7319befc3"
Sep 17 17:09:36.999078 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:09:36.999062    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e10e726e23d06ddfe502a7d0960897b1053e2a67bb943608906fb7319befc3\": container with ID starting with f8e10e726e23d06ddfe502a7d0960897b1053e2a67bb943608906fb7319befc3 not found: ID does not exist" containerID="f8e10e726e23d06ddfe502a7d0960897b1053e2a67bb943608906fb7319befc3"
Sep 17 17:09:36.999116 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.999083    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e10e726e23d06ddfe502a7d0960897b1053e2a67bb943608906fb7319befc3"} err="failed to get container status \"f8e10e726e23d06ddfe502a7d0960897b1053e2a67bb943608906fb7319befc3\": rpc error: code = NotFound desc = could not find container \"f8e10e726e23d06ddfe502a7d0960897b1053e2a67bb943608906fb7319befc3\": container with ID starting with f8e10e726e23d06ddfe502a7d0960897b1053e2a67bb943608906fb7319befc3 not found: ID does not exist"
Sep 17 17:09:36.999116 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.999097    2577 scope.go:117] "RemoveContainer" containerID="af90734ff049a8a7f18d3dffa4dababddbe4d0773da341cbee54041cf4bc49aa"
Sep 17 17:09:36.999329 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:09:36.999312    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af90734ff049a8a7f18d3dffa4dababddbe4d0773da341cbee54041cf4bc49aa\": container with ID starting with af90734ff049a8a7f18d3dffa4dababddbe4d0773da341cbee54041cf4bc49aa not found: ID does not exist" containerID="af90734ff049a8a7f18d3dffa4dababddbe4d0773da341cbee54041cf4bc49aa"
Sep 17 17:09:36.999370 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:36.999334    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af90734ff049a8a7f18d3dffa4dababddbe4d0773da341cbee54041cf4bc49aa"} err="failed to get container status \"af90734ff049a8a7f18d3dffa4dababddbe4d0773da341cbee54041cf4bc49aa\": rpc error: code = NotFound desc = could not find container \"af90734ff049a8a7f18d3dffa4dababddbe4d0773da341cbee54041cf4bc49aa\": container with ID starting with af90734ff049a8a7f18d3dffa4dababddbe4d0773da341cbee54041cf4bc49aa not found: ID does not exist"
Sep 17 17:09:37.002786 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:37.002764    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-4j5zz"]
Sep 17 17:09:37.050579 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:37.050553    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75ede61f-d515-4b5d-bc57-ecbf5188c163-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:09:37.050579 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:37.050580    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75ede61f-d515-4b5d-bc57-ecbf5188c163-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:09:37.050700 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:37.050590    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75ede61f-d515-4b5d-bc57-ecbf5188c163-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:09:37.050700 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:37.050599    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwvgw\" (UniqueName: \"kubernetes.io/projected/75ede61f-d515-4b5d-bc57-ecbf5188c163-kube-api-access-jwvgw\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:09:37.050700 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:37.050607    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75ede61f-d515-4b5d-bc57-ecbf5188c163-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:09:38.195838 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.195805    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"]
Sep 17 17:09:38.196215 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.196143    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kube-rbac-proxy"
Sep 17 17:09:38.196215 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.196168    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kube-rbac-proxy"
Sep 17 17:09:38.196215 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.196182    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="storage-initializer"
Sep 17 17:09:38.196215 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.196187    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="storage-initializer"
Sep 17 17:09:38.196215 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.196194    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kserve-container"
Sep 17 17:09:38.196215 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.196200    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kserve-container"
Sep 17 17:09:38.196439 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.196254    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kserve-container"
Sep 17 17:09:38.196439 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.196261    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" containerName="kube-rbac-proxy"
Sep 17 17:09:38.200615 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.200594    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.203287 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.203246    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:09:38.203417 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.203248    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:09:38.203417 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.203246    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 17:09:38.203417 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.203248    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:09:38.203417 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.203248    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 17:09:38.203622 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.203255    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:09:38.204091 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.204078    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:09:38.210501 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.210473    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"]
Sep 17 17:09:38.228751 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.228731    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"]
Sep 17 17:09:38.231839 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.231825    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.234074 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.234054    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 17:09:38.241346 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.241329    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"]
Sep 17 17:09:38.259451 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.259433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61678897-dc39-4999-ac2f-6d64111fe86d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.259555 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.259464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fggrh\" (UniqueName: \"kubernetes.io/projected/61678897-dc39-4999-ac2f-6d64111fe86d-kube-api-access-fggrh\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.259555 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.259489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-gxr5b\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.259555 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.259528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61678897-dc39-4999-ac2f-6d64111fe86d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.259765 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.259592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-cabundle-cert\") pod \"isvc-canary-promote-predictor-8bc5b857-gxr5b\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.259765 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.259615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8bc5b857-gxr5b\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.259765 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.259662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61678897-dc39-4999-ac2f-6d64111fe86d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.259765 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.259679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61678897-dc39-4999-ac2f-6d64111fe86d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.259765 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.259696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8bc5b857-gxr5b\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.259765 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.259718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnzq5\" (UniqueName: \"kubernetes.io/projected/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kube-api-access-hnzq5\") pod \"isvc-canary-promote-predictor-8bc5b857-gxr5b\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.360790 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.360764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fggrh\" (UniqueName: \"kubernetes.io/projected/61678897-dc39-4999-ac2f-6d64111fe86d-kube-api-access-fggrh\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.360922 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.360799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-gxr5b\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.360922 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.360820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61678897-dc39-4999-ac2f-6d64111fe86d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.360922 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.360869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-cabundle-cert\") pod \"isvc-canary-promote-predictor-8bc5b857-gxr5b\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.360922 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.360914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8bc5b857-gxr5b\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.361147 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.360984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61678897-dc39-4999-ac2f-6d64111fe86d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.361147 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.361012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61678897-dc39-4999-ac2f-6d64111fe86d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.361147 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.361043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8bc5b857-gxr5b\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.361303 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.361239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnzq5\" (UniqueName: \"kubernetes.io/projected/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kube-api-access-hnzq5\") pod \"isvc-canary-promote-predictor-8bc5b857-gxr5b\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.361303 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.361285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61678897-dc39-4999-ac2f-6d64111fe86d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.361425 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:09:38.361403    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 17:09:38.361486 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:09:38.361474    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61678897-dc39-4999-ac2f-6d64111fe86d-proxy-tls podName:61678897-dc39-4999-ac2f-6d64111fe86d nodeName:}" failed. No retries permitted until 2026-09-17 17:09:38.861456582 +0000 UTC m=+1696.886364849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/61678897-dc39-4999-ac2f-6d64111fe86d-proxy-tls") pod "isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" (UID: "61678897-dc39-4999-ac2f-6d64111fe86d") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 17:09:38.361557 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.361502    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61678897-dc39-4999-ac2f-6d64111fe86d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.361848 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.361792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8bc5b857-gxr5b\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.361848 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.361792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-cabundle-cert\") pod \"isvc-canary-promote-predictor-8bc5b857-gxr5b\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.362021 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.361867    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61678897-dc39-4999-ac2f-6d64111fe86d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.362021 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.361874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61678897-dc39-4999-ac2f-6d64111fe86d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.362219 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.362092    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8bc5b857-gxr5b\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.364464 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.364441    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-gxr5b\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.379467 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.379438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fggrh\" (UniqueName: \"kubernetes.io/projected/61678897-dc39-4999-ac2f-6d64111fe86d-kube-api-access-fggrh\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.380461 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.380444    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnzq5\" (UniqueName: \"kubernetes.io/projected/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kube-api-access-hnzq5\") pod \"isvc-canary-promote-predictor-8bc5b857-gxr5b\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.447402 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.447344    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75ede61f-d515-4b5d-bc57-ecbf5188c163" path="/var/lib/kubelet/pods/75ede61f-d515-4b5d-bc57-ecbf5188c163/volumes"
Sep 17 17:09:38.540978 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.540952    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:38.814975 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.814953    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"]
Sep 17 17:09:38.817177 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:09:38.817138    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode815eaa7_5bd5_4bc4_90e2_3a7ff2e87e4a.slice/crio-404ca9570807d7edee8083332ca7731d7666bd9226c575fba3a61cddaec3c2ba WatchSource:0}: Error finding container 404ca9570807d7edee8083332ca7731d7666bd9226c575fba3a61cddaec3c2ba: Status 404 returned error can't find the container with id 404ca9570807d7edee8083332ca7731d7666bd9226c575fba3a61cddaec3c2ba
Sep 17 17:09:38.865196 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.865154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61678897-dc39-4999-ac2f-6d64111fe86d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.867396 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.867377    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61678897-dc39-4999-ac2f-6d64111fe86d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-qhtb6\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:38.986438 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.986403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" event={"ID":"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a","Type":"ContainerStarted","Data":"786f1cb8ed3257fd302e9bb3a0f9cde418fe73766adcf1baeb8080d7fcd5900e"}
Sep 17 17:09:38.986604 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:38.986464    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" event={"ID":"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a","Type":"ContainerStarted","Data":"404ca9570807d7edee8083332ca7731d7666bd9226c575fba3a61cddaec3c2ba"}
Sep 17 17:09:39.111366 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:39.111287    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:39.388534 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:39.388467    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"]
Sep 17 17:09:39.391433 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:09:39.391407    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61678897_dc39_4999_ac2f_6d64111fe86d.slice/crio-4139295ab02d12e04b7923d1d3c6e1a55fc3c37a739312cb8eda889b66a43c3b WatchSource:0}: Error finding container 4139295ab02d12e04b7923d1d3c6e1a55fc3c37a739312cb8eda889b66a43c3b: Status 404 returned error can't find the container with id 4139295ab02d12e04b7923d1d3c6e1a55fc3c37a739312cb8eda889b66a43c3b
Sep 17 17:09:39.993346 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:39.993312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" event={"ID":"61678897-dc39-4999-ac2f-6d64111fe86d","Type":"ContainerStarted","Data":"18703cdb9e8fa31d03eaa888095c10cf344bcd367980961009d9baaa757481c4"}
Sep 17 17:09:39.993527 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:39.993378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" event={"ID":"61678897-dc39-4999-ac2f-6d64111fe86d","Type":"ContainerStarted","Data":"4139295ab02d12e04b7923d1d3c6e1a55fc3c37a739312cb8eda889b66a43c3b"}
Sep 17 17:09:43.009098 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.009069    2577 generic.go:356] "Generic (PLEG): container finished" podID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerID="786f1cb8ed3257fd302e9bb3a0f9cde418fe73766adcf1baeb8080d7fcd5900e" exitCode=0
Sep 17 17:09:43.009445 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.009148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" event={"ID":"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a","Type":"ContainerDied","Data":"786f1cb8ed3257fd302e9bb3a0f9cde418fe73766adcf1baeb8080d7fcd5900e"}
Sep 17 17:09:43.385810 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.385770    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"]
Sep 17 17:09:43.389344 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.389322    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:43.392153 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.392131    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 17 17:09:43.392278 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.392181    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 17:09:43.398001 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.397957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea96907f-306e-4754-a631-a6a04a6f71bd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:43.398118 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.398008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea96907f-306e-4754-a631-a6a04a6f71bd-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:43.398118 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.398036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea96907f-306e-4754-a631-a6a04a6f71bd-proxy-tls\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:43.398118 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.398088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea96907f-306e-4754-a631-a6a04a6f71bd-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:43.398118 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.398116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7cg8\" (UniqueName: \"kubernetes.io/projected/ea96907f-306e-4754-a631-a6a04a6f71bd-kube-api-access-x7cg8\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:43.399848 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.399828    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"]
Sep 17 17:09:43.498929 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.498870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea96907f-306e-4754-a631-a6a04a6f71bd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:43.498929 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.498918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea96907f-306e-4754-a631-a6a04a6f71bd-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:43.499055 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.499037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea96907f-306e-4754-a631-a6a04a6f71bd-proxy-tls\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:43.499121 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.499104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea96907f-306e-4754-a631-a6a04a6f71bd-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:43.499201 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.499144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7cg8\" (UniqueName: \"kubernetes.io/projected/ea96907f-306e-4754-a631-a6a04a6f71bd-kube-api-access-x7cg8\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:43.499282 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:09:43.499268    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 17 17:09:43.499334 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.499310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea96907f-306e-4754-a631-a6a04a6f71bd-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:43.499334 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:09:43.499329    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea96907f-306e-4754-a631-a6a04a6f71bd-proxy-tls podName:ea96907f-306e-4754-a631-a6a04a6f71bd nodeName:}" failed. No retries permitted until 2026-09-17 17:09:43.999311317 +0000 UTC m=+1702.024219587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ea96907f-306e-4754-a631-a6a04a6f71bd-proxy-tls") pod "isvc-sklearn-v2-predictor-774579cd8d-rz5ld" (UID: "ea96907f-306e-4754-a631-a6a04a6f71bd") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 17 17:09:43.499584 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.499564    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea96907f-306e-4754-a631-a6a04a6f71bd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:43.499767 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.499745    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea96907f-306e-4754-a631-a6a04a6f71bd-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:43.508248 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:43.508226    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7cg8\" (UniqueName: \"kubernetes.io/projected/ea96907f-306e-4754-a631-a6a04a6f71bd-kube-api-access-x7cg8\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:44.003179 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:44.003127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea96907f-306e-4754-a631-a6a04a6f71bd-proxy-tls\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:44.005669 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:44.005639    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea96907f-306e-4754-a631-a6a04a6f71bd-proxy-tls\") pod \"isvc-sklearn-v2-predictor-774579cd8d-rz5ld\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:44.016016 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:44.015984    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" event={"ID":"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a","Type":"ContainerStarted","Data":"63d539e553fa0dbad4d8dc7fa9fbfb2d8217fadbcf2e214902f680c76cf02ef0"}
Sep 17 17:09:44.016433 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:44.016046    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" event={"ID":"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a","Type":"ContainerStarted","Data":"300b75b51149a70935d4cc6de74bfd7fe701a94704ec7ac2d93f89de9f3af871"}
Sep 17 17:09:44.016433 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:44.016410    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:44.017520 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:44.017499    2577 generic.go:356] "Generic (PLEG): container finished" podID="61678897-dc39-4999-ac2f-6d64111fe86d" containerID="18703cdb9e8fa31d03eaa888095c10cf344bcd367980961009d9baaa757481c4" exitCode=0
Sep 17 17:09:44.017632 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:44.017571    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" event={"ID":"61678897-dc39-4999-ac2f-6d64111fe86d","Type":"ContainerDied","Data":"18703cdb9e8fa31d03eaa888095c10cf344bcd367980961009d9baaa757481c4"}
Sep 17 17:09:44.036420 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:44.036361    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" podStartSLOduration=2.013879339 podStartE2EDuration="6.036347055s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.022467716s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:09:44.035145776 +0000 UTC m=+1702.060054264" watchObservedRunningTime="2026-09-17 17:09:44.036347055 +0000 UTC m=+1702.061255411"
Sep 17 17:09:44.299296 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:44.299273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:44.573317 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:44.573293    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"]
Sep 17 17:09:44.575257 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:09:44.575232    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea96907f_306e_4754_a631_a6a04a6f71bd.slice/crio-2465d21414d6e28fe511522114c895a28a5c3a08f0ceba18a4661a996f6e9def WatchSource:0}: Error finding container 2465d21414d6e28fe511522114c895a28a5c3a08f0ceba18a4661a996f6e9def: Status 404 returned error can't find the container with id 2465d21414d6e28fe511522114c895a28a5c3a08f0ceba18a4661a996f6e9def
Sep 17 17:09:45.025246 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:45.025127    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" event={"ID":"61678897-dc39-4999-ac2f-6d64111fe86d","Type":"ContainerStarted","Data":"186fbf0c4b64d1d4586b7dd085cde8b9b82ec2b68c0a9727f3979715dcdf9d24"}
Sep 17 17:09:45.025246 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:45.025216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" event={"ID":"61678897-dc39-4999-ac2f-6d64111fe86d","Type":"ContainerStarted","Data":"4247c940045f90a4cc806e8b0b7dec101472c40262f357bc4fe87118ea18a74f"}
Sep 17 17:09:45.025690 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:45.025549    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:45.025690 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:45.025616    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:45.027099 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:45.027072    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" event={"ID":"ea96907f-306e-4754-a631-a6a04a6f71bd","Type":"ContainerStarted","Data":"85cd85eb436cc1be59923714b47e6aa6a79ea59868bbbf153a87c6d7669ea436"}
Sep 17 17:09:45.027257 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:45.027129    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" event={"ID":"ea96907f-306e-4754-a631-a6a04a6f71bd","Type":"ContainerStarted","Data":"2465d21414d6e28fe511522114c895a28a5c3a08f0ceba18a4661a996f6e9def"}
Sep 17 17:09:45.027466 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:45.027444    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:45.027633 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:45.027611    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 17:09:45.028550 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:45.028520    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 17:09:45.045855 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:45.045817    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" podStartSLOduration=3.395354573 podStartE2EDuration="7.045799367s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.650444794s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:09:45.044764828 +0000 UTC m=+1703.069673138" watchObservedRunningTime="2026-09-17 17:09:45.045799367 +0000 UTC m=+1703.070707668"
Sep 17 17:09:46.031592 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:46.031544    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 17:09:46.032051 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:46.031636    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 17:09:49.048125 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:49.048093    2577 generic.go:356] "Generic (PLEG): container finished" podID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerID="85cd85eb436cc1be59923714b47e6aa6a79ea59868bbbf153a87c6d7669ea436" exitCode=0
Sep 17 17:09:49.048555 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:49.048131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" event={"ID":"ea96907f-306e-4754-a631-a6a04a6f71bd","Type":"ContainerDied","Data":"85cd85eb436cc1be59923714b47e6aa6a79ea59868bbbf153a87c6d7669ea436"}
Sep 17 17:09:50.060354 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:50.060318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" event={"ID":"ea96907f-306e-4754-a631-a6a04a6f71bd","Type":"ContainerStarted","Data":"9a64af9c604dcb456e227786fc1048450f24b03ec4a7cb05f2f45044ff6b7254"}
Sep 17 17:09:50.060728 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:50.060400    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" event={"ID":"ea96907f-306e-4754-a631-a6a04a6f71bd","Type":"ContainerStarted","Data":"8a3098a51fa2502d1697fbd25b730934491c92e3736677b89d46dd0f9087673e"}
Sep 17 17:09:50.060728 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:50.060691    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:50.080889 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:50.080846    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" podStartSLOduration=3.324671361 podStartE2EDuration="7.080832459s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.756161098s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:09:50.079353209 +0000 UTC m=+1708.104261499" watchObservedRunningTime="2026-09-17 17:09:50.080832459 +0000 UTC m=+1708.105740748"
Sep 17 17:09:51.036931 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:51.036901    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:09:51.037503 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:51.037476    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:09:51.037641 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:51.037528    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 17:09:51.038027 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:51.038003    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 17:09:51.065469 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:51.065442    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:51.066459 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:51.066435    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 17:09:52.070145 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:52.070094    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 17:09:57.075192 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:57.075144    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:09:57.075621 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:09:57.075591    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 17:10:01.038318 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:01.038281    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 17:10:01.038711 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:01.038284    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 17:10:07.076450 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:07.076410    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 17:10:11.037848 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:11.037801    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 17:10:11.038247 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:11.037947    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 17:10:17.076295 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:17.076245    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 17:10:21.038236 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:21.038197    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 17:10:21.038627 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:21.038197    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 17:10:27.075709 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:27.075669    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 17:10:31.037520 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:31.037481    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 17:10:31.037991 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:31.037903    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 17:10:37.076438 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:37.076389    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 17:10:41.038442 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:41.038402    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 17:10:41.038815 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:41.038414    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 17:10:47.075889 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:47.075850    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 17:10:51.037536 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:51.037495    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 17:10:51.037925 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:51.037899    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 17:10:57.076415 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:10:57.076372    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 17:11:00.442183 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:00.442118    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 17:11:01.038312 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:01.038282    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:11:01.038526 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:01.038367    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:11:01.171567 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:01.171534    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"]
Sep 17 17:11:01.389838 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:01.389801    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kube-rbac-proxy" containerID="cri-o://63d539e553fa0dbad4d8dc7fa9fbfb2d8217fadbcf2e214902f680c76cf02ef0" gracePeriod=30
Sep 17 17:11:01.390052 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:01.389790    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kserve-container" containerID="cri-o://300b75b51149a70935d4cc6de74bfd7fe701a94704ec7ac2d93f89de9f3af871" gracePeriod=30
Sep 17 17:11:02.396059 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:02.396023    2577 generic.go:356] "Generic (PLEG): container finished" podID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerID="63d539e553fa0dbad4d8dc7fa9fbfb2d8217fadbcf2e214902f680c76cf02ef0" exitCode=2
Sep 17 17:11:02.396518 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:02.396056    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" event={"ID":"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a","Type":"ContainerDied","Data":"63d539e553fa0dbad4d8dc7fa9fbfb2d8217fadbcf2e214902f680c76cf02ef0"}
Sep 17 17:11:06.031852 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:06.031810    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 17:11:07.627377 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.627354    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:11:07.685472 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.685417    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnzq5\" (UniqueName: \"kubernetes.io/projected/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kube-api-access-hnzq5\") pod \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") "
Sep 17 17:11:07.685472 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.685450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-proxy-tls\") pod \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") "
Sep 17 17:11:07.685610 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.685491    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kserve-provision-location\") pod \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") "
Sep 17 17:11:07.685610 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.685539    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-cabundle-cert\") pod \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") "
Sep 17 17:11:07.685610 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.685587    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kube-rbac-proxy-sar-config\") pod \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\" (UID: \"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a\") "
Sep 17 17:11:07.685858 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.685829    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" (UID: "e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:11:07.685967 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.685945    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" (UID: "e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:11:07.686046 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.686014    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" (UID: "e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:11:07.687527 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.687506    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kube-api-access-hnzq5" (OuterVolumeSpecName: "kube-api-access-hnzq5") pod "e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" (UID: "e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a"). InnerVolumeSpecName "kube-api-access-hnzq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:11:07.687589 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.687575    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" (UID: "e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:11:07.786385 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.786348    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnzq5\" (UniqueName: \"kubernetes.io/projected/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kube-api-access-hnzq5\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:11:07.786486 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.786392    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:11:07.786486 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.786411    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:11:07.786486 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.786430    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:11:07.786486 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:07.786454    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:11:08.424992 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:08.424957    2577 generic.go:356] "Generic (PLEG): container finished" podID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerID="300b75b51149a70935d4cc6de74bfd7fe701a94704ec7ac2d93f89de9f3af871" exitCode=0
Sep 17 17:11:08.425144 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:08.425026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" event={"ID":"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a","Type":"ContainerDied","Data":"300b75b51149a70935d4cc6de74bfd7fe701a94704ec7ac2d93f89de9f3af871"}
Sep 17 17:11:08.425144 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:08.425111    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b" event={"ID":"e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a","Type":"ContainerDied","Data":"404ca9570807d7edee8083332ca7731d7666bd9226c575fba3a61cddaec3c2ba"}
Sep 17 17:11:08.425144 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:08.425034    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"
Sep 17 17:11:08.425144 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:08.425141    2577 scope.go:117] "RemoveContainer" containerID="63d539e553fa0dbad4d8dc7fa9fbfb2d8217fadbcf2e214902f680c76cf02ef0"
Sep 17 17:11:08.433722 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:08.433702    2577 scope.go:117] "RemoveContainer" containerID="300b75b51149a70935d4cc6de74bfd7fe701a94704ec7ac2d93f89de9f3af871"
Sep 17 17:11:08.440907 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:08.440892    2577 scope.go:117] "RemoveContainer" containerID="786f1cb8ed3257fd302e9bb3a0f9cde418fe73766adcf1baeb8080d7fcd5900e"
Sep 17 17:11:08.447989 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:08.447970    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"]
Sep 17 17:11:08.448250 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:08.448234    2577 scope.go:117] "RemoveContainer" containerID="63d539e553fa0dbad4d8dc7fa9fbfb2d8217fadbcf2e214902f680c76cf02ef0"
Sep 17 17:11:08.448494 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:11:08.448476    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d539e553fa0dbad4d8dc7fa9fbfb2d8217fadbcf2e214902f680c76cf02ef0\": container with ID starting with 63d539e553fa0dbad4d8dc7fa9fbfb2d8217fadbcf2e214902f680c76cf02ef0 not found: ID does not exist" containerID="63d539e553fa0dbad4d8dc7fa9fbfb2d8217fadbcf2e214902f680c76cf02ef0"
Sep 17 17:11:08.448558 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:08.448502    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d539e553fa0dbad4d8dc7fa9fbfb2d8217fadbcf2e214902f680c76cf02ef0"} err="failed to get container status \"63d539e553fa0dbad4d8dc7fa9fbfb2d8217fadbcf2e214902f680c76cf02ef0\": rpc error: code = NotFound desc = could not find container \"63d539e553fa0dbad4d8dc7fa9fbfb2d8217fadbcf2e214902f680c76cf02ef0\": container with ID starting with 63d539e553fa0dbad4d8dc7fa9fbfb2d8217fadbcf2e214902f680c76cf02ef0 not found: ID does not exist"
Sep 17 17:11:08.448558 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:08.448518    2577 scope.go:117] "RemoveContainer" containerID="300b75b51149a70935d4cc6de74bfd7fe701a94704ec7ac2d93f89de9f3af871"
Sep 17 17:11:08.448790 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:11:08.448774    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300b75b51149a70935d4cc6de74bfd7fe701a94704ec7ac2d93f89de9f3af871\": container with ID starting with 300b75b51149a70935d4cc6de74bfd7fe701a94704ec7ac2d93f89de9f3af871 not found: ID does not exist" containerID="300b75b51149a70935d4cc6de74bfd7fe701a94704ec7ac2d93f89de9f3af871"
Sep 17 17:11:08.448844 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:08.448794    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300b75b51149a70935d4cc6de74bfd7fe701a94704ec7ac2d93f89de9f3af871"} err="failed to get container status \"300b75b51149a70935d4cc6de74bfd7fe701a94704ec7ac2d93f89de9f3af871\": rpc error: code = NotFound desc = could not find container \"300b75b51149a70935d4cc6de74bfd7fe701a94704ec7ac2d93f89de9f3af871\": container with ID starting with 300b75b51149a70935d4cc6de74bfd7fe701a94704ec7ac2d93f89de9f3af871 not found: ID does not exist"
Sep 17 17:11:08.448844 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:08.448816    2577 scope.go:117] "RemoveContainer" containerID="786f1cb8ed3257fd302e9bb3a0f9cde418fe73766adcf1baeb8080d7fcd5900e"
Sep 17 17:11:08.449019 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:11:08.449000    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786f1cb8ed3257fd302e9bb3a0f9cde418fe73766adcf1baeb8080d7fcd5900e\": container with ID starting with 786f1cb8ed3257fd302e9bb3a0f9cde418fe73766adcf1baeb8080d7fcd5900e not found: ID does not exist" containerID="786f1cb8ed3257fd302e9bb3a0f9cde418fe73766adcf1baeb8080d7fcd5900e"
Sep 17 17:11:08.449066 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:08.449025    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786f1cb8ed3257fd302e9bb3a0f9cde418fe73766adcf1baeb8080d7fcd5900e"} err="failed to get container status \"786f1cb8ed3257fd302e9bb3a0f9cde418fe73766adcf1baeb8080d7fcd5900e\": rpc error: code = NotFound desc = could not find container \"786f1cb8ed3257fd302e9bb3a0f9cde418fe73766adcf1baeb8080d7fcd5900e\": container with ID starting with 786f1cb8ed3257fd302e9bb3a0f9cde418fe73766adcf1baeb8080d7fcd5900e not found: ID does not exist"
Sep 17 17:11:08.452024 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:08.452005    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gxr5b"]
Sep 17 17:11:10.447365 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:10.447330    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" path="/var/lib/kubelet/pods/e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a/volumes"
Sep 17 17:11:10.447743 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:10.447715    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:11:13.518741 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:13.514527    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"]
Sep 17 17:11:13.518741 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:13.514940    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kserve-container" containerID="cri-o://8a3098a51fa2502d1697fbd25b730934491c92e3736677b89d46dd0f9087673e" gracePeriod=30
Sep 17 17:11:13.518741 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:13.515249    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kube-rbac-proxy" containerID="cri-o://9a64af9c604dcb456e227786fc1048450f24b03ec4a7cb05f2f45044ff6b7254" gracePeriod=30
Sep 17 17:11:14.457400 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:14.457366    2577 generic.go:356] "Generic (PLEG): container finished" podID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerID="9a64af9c604dcb456e227786fc1048450f24b03ec4a7cb05f2f45044ff6b7254" exitCode=2
Sep 17 17:11:14.457552 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:14.457441    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" event={"ID":"ea96907f-306e-4754-a631-a6a04a6f71bd","Type":"ContainerDied","Data":"9a64af9c604dcb456e227786fc1048450f24b03ec4a7cb05f2f45044ff6b7254"}
Sep 17 17:11:17.071130 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.071045    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 17 17:11:17.471928 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.471902    2577 generic.go:356] "Generic (PLEG): container finished" podID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerID="8a3098a51fa2502d1697fbd25b730934491c92e3736677b89d46dd0f9087673e" exitCode=0
Sep 17 17:11:17.472066 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.471975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" event={"ID":"ea96907f-306e-4754-a631-a6a04a6f71bd","Type":"ContainerDied","Data":"8a3098a51fa2502d1697fbd25b730934491c92e3736677b89d46dd0f9087673e"}
Sep 17 17:11:17.472066 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.472043    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld" event={"ID":"ea96907f-306e-4754-a631-a6a04a6f71bd","Type":"ContainerDied","Data":"2465d21414d6e28fe511522114c895a28a5c3a08f0ceba18a4661a996f6e9def"}
Sep 17 17:11:17.472174 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.472074    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2465d21414d6e28fe511522114c895a28a5c3a08f0ceba18a4661a996f6e9def"
Sep 17 17:11:17.474173 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.474148    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:11:17.556530 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.556503    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea96907f-306e-4754-a631-a6a04a6f71bd-kube-rbac-proxy-sar-config\") pod \"ea96907f-306e-4754-a631-a6a04a6f71bd\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") "
Sep 17 17:11:17.556530 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.556531    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea96907f-306e-4754-a631-a6a04a6f71bd-cabundle-cert\") pod \"ea96907f-306e-4754-a631-a6a04a6f71bd\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") "
Sep 17 17:11:17.556706 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.556555    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7cg8\" (UniqueName: \"kubernetes.io/projected/ea96907f-306e-4754-a631-a6a04a6f71bd-kube-api-access-x7cg8\") pod \"ea96907f-306e-4754-a631-a6a04a6f71bd\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") "
Sep 17 17:11:17.556706 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.556598    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea96907f-306e-4754-a631-a6a04a6f71bd-kserve-provision-location\") pod \"ea96907f-306e-4754-a631-a6a04a6f71bd\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") "
Sep 17 17:11:17.556706 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.556624    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea96907f-306e-4754-a631-a6a04a6f71bd-proxy-tls\") pod \"ea96907f-306e-4754-a631-a6a04a6f71bd\" (UID: \"ea96907f-306e-4754-a631-a6a04a6f71bd\") "
Sep 17 17:11:17.556902 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.556872    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea96907f-306e-4754-a631-a6a04a6f71bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea96907f-306e-4754-a631-a6a04a6f71bd" (UID: "ea96907f-306e-4754-a631-a6a04a6f71bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:11:17.557001 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.556895    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea96907f-306e-4754-a631-a6a04a6f71bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ea96907f-306e-4754-a631-a6a04a6f71bd" (UID: "ea96907f-306e-4754-a631-a6a04a6f71bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:11:17.557001 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.556919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea96907f-306e-4754-a631-a6a04a6f71bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea96907f-306e-4754-a631-a6a04a6f71bd" (UID: "ea96907f-306e-4754-a631-a6a04a6f71bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:11:17.557116 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.557014    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea96907f-306e-4754-a631-a6a04a6f71bd-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:11:17.557116 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.557032    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea96907f-306e-4754-a631-a6a04a6f71bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:11:17.557116 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.557044    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea96907f-306e-4754-a631-a6a04a6f71bd-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:11:17.558587 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.558566    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea96907f-306e-4754-a631-a6a04a6f71bd-kube-api-access-x7cg8" (OuterVolumeSpecName: "kube-api-access-x7cg8") pod "ea96907f-306e-4754-a631-a6a04a6f71bd" (UID: "ea96907f-306e-4754-a631-a6a04a6f71bd"). InnerVolumeSpecName "kube-api-access-x7cg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:11:17.558672 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.558617    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea96907f-306e-4754-a631-a6a04a6f71bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea96907f-306e-4754-a631-a6a04a6f71bd" (UID: "ea96907f-306e-4754-a631-a6a04a6f71bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:11:17.657977 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.657926    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7cg8\" (UniqueName: \"kubernetes.io/projected/ea96907f-306e-4754-a631-a6a04a6f71bd-kube-api-access-x7cg8\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:11:17.657977 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:17.657946    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea96907f-306e-4754-a631-a6a04a6f71bd-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:11:18.476739 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.476709    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"
Sep 17 17:11:18.495205 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.495176    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"]
Sep 17 17:11:18.497189 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.497155    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-774579cd8d-rz5ld"]
Sep 17 17:11:18.681241 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681209    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"]
Sep 17 17:11:18.681591 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681575    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kserve-container"
Sep 17 17:11:18.681683 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681593    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kserve-container"
Sep 17 17:11:18.681683 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681609    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="storage-initializer"
Sep 17 17:11:18.681683 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681617    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="storage-initializer"
Sep 17 17:11:18.681683 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681628    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kserve-container"
Sep 17 17:11:18.681683 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681636    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kserve-container"
Sep 17 17:11:18.681683 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681647    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kube-rbac-proxy"
Sep 17 17:11:18.681683 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681655    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kube-rbac-proxy"
Sep 17 17:11:18.681683 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681674    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="storage-initializer"
Sep 17 17:11:18.681683 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681682    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="storage-initializer"
Sep 17 17:11:18.682128 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681699    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kube-rbac-proxy"
Sep 17 17:11:18.682128 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681708    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kube-rbac-proxy"
Sep 17 17:11:18.682128 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681784    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kube-rbac-proxy"
Sep 17 17:11:18.682128 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681795    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" containerName="kserve-container"
Sep 17 17:11:18.682128 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681806    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kube-rbac-proxy"
Sep 17 17:11:18.682128 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.681819    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e815eaa7-5bd5-4bc4-90e2-3a7ff2e87e4a" containerName="kserve-container"
Sep 17 17:11:18.685281 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.685260    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.687785 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.687766    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 17 17:11:18.687987 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.687973    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 17 17:11:18.695289 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.695270    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"]
Sep 17 17:11:18.764391 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.764344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f975d9a2-3308-4893-80d3-27e16035fa56-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.764391 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.764377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f975d9a2-3308-4893-80d3-27e16035fa56-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.764531 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.764407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fzpg\" (UniqueName: \"kubernetes.io/projected/f975d9a2-3308-4893-80d3-27e16035fa56-kube-api-access-4fzpg\") pod \"isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.764531 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.764432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f975d9a2-3308-4893-80d3-27e16035fa56-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.764531 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.764498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f975d9a2-3308-4893-80d3-27e16035fa56-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.865758 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.865730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fzpg\" (UniqueName: \"kubernetes.io/projected/f975d9a2-3308-4893-80d3-27e16035fa56-kube-api-access-4fzpg\") pod \"isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.865865 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.865761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f975d9a2-3308-4893-80d3-27e16035fa56-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.865865 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.865789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f975d9a2-3308-4893-80d3-27e16035fa56-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.865865 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.865812    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f975d9a2-3308-4893-80d3-27e16035fa56-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.865865 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.865846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f975d9a2-3308-4893-80d3-27e16035fa56-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.866224 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.866202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f975d9a2-3308-4893-80d3-27e16035fa56-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.866488 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.866462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f975d9a2-3308-4893-80d3-27e16035fa56-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.866601 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.866542    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f975d9a2-3308-4893-80d3-27e16035fa56-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.868334 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.868314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f975d9a2-3308-4893-80d3-27e16035fa56-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.874073 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.874050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fzpg\" (UniqueName: \"kubernetes.io/projected/f975d9a2-3308-4893-80d3-27e16035fa56-kube-api-access-4fzpg\") pod \"isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:18.995008 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:18.994986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:19.264926 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:19.264900    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"]
Sep 17 17:11:19.267377 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:11:19.267353    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf975d9a2_3308_4893_80d3_27e16035fa56.slice/crio-2ff92878aad28192276b274d8e17e21f86962356deeef470e4afd987d03589a1 WatchSource:0}: Error finding container 2ff92878aad28192276b274d8e17e21f86962356deeef470e4afd987d03589a1: Status 404 returned error can't find the container with id 2ff92878aad28192276b274d8e17e21f86962356deeef470e4afd987d03589a1
Sep 17 17:11:19.482122 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:19.482082    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" event={"ID":"f975d9a2-3308-4893-80d3-27e16035fa56","Type":"ContainerStarted","Data":"a9dd2d3c7dda98a32b2850833eaf39b1c1ddfeb4c6b9c3b603595f0388e8f288"}
Sep 17 17:11:19.482596 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:19.482144    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" event={"ID":"f975d9a2-3308-4893-80d3-27e16035fa56","Type":"ContainerStarted","Data":"2ff92878aad28192276b274d8e17e21f86962356deeef470e4afd987d03589a1"}
Sep 17 17:11:20.446822 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:20.446788    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea96907f-306e-4754-a631-a6a04a6f71bd" path="/var/lib/kubelet/pods/ea96907f-306e-4754-a631-a6a04a6f71bd/volumes"
Sep 17 17:11:22.532016 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:22.531985    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 17:11:22.533328 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:22.533304    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 17:11:23.502875 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:23.502840    2577 generic.go:356] "Generic (PLEG): container finished" podID="f975d9a2-3308-4893-80d3-27e16035fa56" containerID="a9dd2d3c7dda98a32b2850833eaf39b1c1ddfeb4c6b9c3b603595f0388e8f288" exitCode=0
Sep 17 17:11:23.502996 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:23.502917    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" event={"ID":"f975d9a2-3308-4893-80d3-27e16035fa56","Type":"ContainerDied","Data":"a9dd2d3c7dda98a32b2850833eaf39b1c1ddfeb4c6b9c3b603595f0388e8f288"}
Sep 17 17:11:24.510170 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:24.510115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" event={"ID":"f975d9a2-3308-4893-80d3-27e16035fa56","Type":"ContainerStarted","Data":"2017d58d3a7b53394f36932227ad2031c2ac02411f003b303b7e70a1e00bf736"}
Sep 17 17:11:24.510770 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:24.510201    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" event={"ID":"f975d9a2-3308-4893-80d3-27e16035fa56","Type":"ContainerStarted","Data":"2f8a5e207d2af667b7885e52f95ae8cf5958c83935390a0a35555612036cd494"}
Sep 17 17:11:24.510770 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:24.510395    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:24.528455 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:24.528409    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" podStartSLOduration=2.555080159 podStartE2EDuration="6.528394525s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.973314366s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:11:24.527273164 +0000 UTC m=+1802.552181452" watchObservedRunningTime="2026-09-17 17:11:24.528394525 +0000 UTC m=+1802.553302812"
Sep 17 17:11:25.515119 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:25.515079    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:25.516465 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:25.516435    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 17:11:26.519775 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:26.519732    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 17:11:31.524101 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:31.524065    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:11:31.524554 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:31.524526    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 17:11:41.525312 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:41.525271    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 17:11:51.524519 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:11:51.524477    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 17:12:01.525363 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:01.525325    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 17:12:11.525358 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:11.525319    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 17:12:21.525490 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:21.525452    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 17:12:31.524880 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:31.524842    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 17:12:38.451943 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:38.451908    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"]
Sep 17 17:12:38.452423 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:38.452188    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kserve-container" containerID="cri-o://4247c940045f90a4cc806e8b0b7dec101472c40262f357bc4fe87118ea18a74f" gracePeriod=30
Sep 17 17:12:38.452423 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:38.452244    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kube-rbac-proxy" containerID="cri-o://186fbf0c4b64d1d4586b7dd085cde8b9b82ec2b68c0a9727f3979715dcdf9d24" gracePeriod=30
Sep 17 17:12:38.849443 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:38.849409    2577 generic.go:356] "Generic (PLEG): container finished" podID="61678897-dc39-4999-ac2f-6d64111fe86d" containerID="186fbf0c4b64d1d4586b7dd085cde8b9b82ec2b68c0a9727f3979715dcdf9d24" exitCode=2
Sep 17 17:12:38.849605 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:38.849476    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" event={"ID":"61678897-dc39-4999-ac2f-6d64111fe86d","Type":"ContainerDied","Data":"186fbf0c4b64d1d4586b7dd085cde8b9b82ec2b68c0a9727f3979715dcdf9d24"}
Sep 17 17:12:41.032520 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:41.032483    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 17 17:12:41.037724 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:41.037698    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 17:12:41.525910 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:41.525874    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:12:42.593103 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.593080    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:12:42.758060 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.757996    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61678897-dc39-4999-ac2f-6d64111fe86d-cabundle-cert\") pod \"61678897-dc39-4999-ac2f-6d64111fe86d\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") "
Sep 17 17:12:42.758060 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.758034    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61678897-dc39-4999-ac2f-6d64111fe86d-proxy-tls\") pod \"61678897-dc39-4999-ac2f-6d64111fe86d\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") "
Sep 17 17:12:42.758060 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.758057    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61678897-dc39-4999-ac2f-6d64111fe86d-kube-rbac-proxy-sar-config\") pod \"61678897-dc39-4999-ac2f-6d64111fe86d\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") "
Sep 17 17:12:42.758345 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.758095    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fggrh\" (UniqueName: \"kubernetes.io/projected/61678897-dc39-4999-ac2f-6d64111fe86d-kube-api-access-fggrh\") pod \"61678897-dc39-4999-ac2f-6d64111fe86d\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") "
Sep 17 17:12:42.758345 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.758185    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61678897-dc39-4999-ac2f-6d64111fe86d-kserve-provision-location\") pod \"61678897-dc39-4999-ac2f-6d64111fe86d\" (UID: \"61678897-dc39-4999-ac2f-6d64111fe86d\") "
Sep 17 17:12:42.758481 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.758436    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61678897-dc39-4999-ac2f-6d64111fe86d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "61678897-dc39-4999-ac2f-6d64111fe86d" (UID: "61678897-dc39-4999-ac2f-6d64111fe86d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:12:42.758534 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.758464    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61678897-dc39-4999-ac2f-6d64111fe86d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "61678897-dc39-4999-ac2f-6d64111fe86d" (UID: "61678897-dc39-4999-ac2f-6d64111fe86d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:12:42.758534 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.758480    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61678897-dc39-4999-ac2f-6d64111fe86d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61678897-dc39-4999-ac2f-6d64111fe86d" (UID: "61678897-dc39-4999-ac2f-6d64111fe86d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:12:42.760143 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.760125    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61678897-dc39-4999-ac2f-6d64111fe86d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "61678897-dc39-4999-ac2f-6d64111fe86d" (UID: "61678897-dc39-4999-ac2f-6d64111fe86d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:12:42.760288 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.760271    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61678897-dc39-4999-ac2f-6d64111fe86d-kube-api-access-fggrh" (OuterVolumeSpecName: "kube-api-access-fggrh") pod "61678897-dc39-4999-ac2f-6d64111fe86d" (UID: "61678897-dc39-4999-ac2f-6d64111fe86d"). InnerVolumeSpecName "kube-api-access-fggrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:12:42.858692 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.858672    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61678897-dc39-4999-ac2f-6d64111fe86d-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:12:42.858692 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.858693    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61678897-dc39-4999-ac2f-6d64111fe86d-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:12:42.858862 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.858708    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61678897-dc39-4999-ac2f-6d64111fe86d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:12:42.858862 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.858720    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fggrh\" (UniqueName: \"kubernetes.io/projected/61678897-dc39-4999-ac2f-6d64111fe86d-kube-api-access-fggrh\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:12:42.858862 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.858732    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61678897-dc39-4999-ac2f-6d64111fe86d-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:12:42.872926 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.872898    2577 generic.go:356] "Generic (PLEG): container finished" podID="61678897-dc39-4999-ac2f-6d64111fe86d" containerID="4247c940045f90a4cc806e8b0b7dec101472c40262f357bc4fe87118ea18a74f" exitCode=0
Sep 17 17:12:42.873013 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.872946    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"
Sep 17 17:12:42.873013 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.872985    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" event={"ID":"61678897-dc39-4999-ac2f-6d64111fe86d","Type":"ContainerDied","Data":"4247c940045f90a4cc806e8b0b7dec101472c40262f357bc4fe87118ea18a74f"}
Sep 17 17:12:42.873101 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.873051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6" event={"ID":"61678897-dc39-4999-ac2f-6d64111fe86d","Type":"ContainerDied","Data":"4139295ab02d12e04b7923d1d3c6e1a55fc3c37a739312cb8eda889b66a43c3b"}
Sep 17 17:12:42.873101 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.873078    2577 scope.go:117] "RemoveContainer" containerID="186fbf0c4b64d1d4586b7dd085cde8b9b82ec2b68c0a9727f3979715dcdf9d24"
Sep 17 17:12:42.881319 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.881298    2577 scope.go:117] "RemoveContainer" containerID="4247c940045f90a4cc806e8b0b7dec101472c40262f357bc4fe87118ea18a74f"
Sep 17 17:12:42.891361 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.891345    2577 scope.go:117] "RemoveContainer" containerID="18703cdb9e8fa31d03eaa888095c10cf344bcd367980961009d9baaa757481c4"
Sep 17 17:12:42.897104 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.897082    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"]
Sep 17 17:12:42.899445 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.899430    2577 scope.go:117] "RemoveContainer" containerID="186fbf0c4b64d1d4586b7dd085cde8b9b82ec2b68c0a9727f3979715dcdf9d24"
Sep 17 17:12:42.899670 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:12:42.899651    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186fbf0c4b64d1d4586b7dd085cde8b9b82ec2b68c0a9727f3979715dcdf9d24\": container with ID starting with 186fbf0c4b64d1d4586b7dd085cde8b9b82ec2b68c0a9727f3979715dcdf9d24 not found: ID does not exist" containerID="186fbf0c4b64d1d4586b7dd085cde8b9b82ec2b68c0a9727f3979715dcdf9d24"
Sep 17 17:12:42.899740 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.899681    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186fbf0c4b64d1d4586b7dd085cde8b9b82ec2b68c0a9727f3979715dcdf9d24"} err="failed to get container status \"186fbf0c4b64d1d4586b7dd085cde8b9b82ec2b68c0a9727f3979715dcdf9d24\": rpc error: code = NotFound desc = could not find container \"186fbf0c4b64d1d4586b7dd085cde8b9b82ec2b68c0a9727f3979715dcdf9d24\": container with ID starting with 186fbf0c4b64d1d4586b7dd085cde8b9b82ec2b68c0a9727f3979715dcdf9d24 not found: ID does not exist"
Sep 17 17:12:42.899740 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.899705    2577 scope.go:117] "RemoveContainer" containerID="4247c940045f90a4cc806e8b0b7dec101472c40262f357bc4fe87118ea18a74f"
Sep 17 17:12:42.899956 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:12:42.899938    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4247c940045f90a4cc806e8b0b7dec101472c40262f357bc4fe87118ea18a74f\": container with ID starting with 4247c940045f90a4cc806e8b0b7dec101472c40262f357bc4fe87118ea18a74f not found: ID does not exist" containerID="4247c940045f90a4cc806e8b0b7dec101472c40262f357bc4fe87118ea18a74f"
Sep 17 17:12:42.899996 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.899963    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4247c940045f90a4cc806e8b0b7dec101472c40262f357bc4fe87118ea18a74f"} err="failed to get container status \"4247c940045f90a4cc806e8b0b7dec101472c40262f357bc4fe87118ea18a74f\": rpc error: code = NotFound desc = could not find container \"4247c940045f90a4cc806e8b0b7dec101472c40262f357bc4fe87118ea18a74f\": container with ID starting with 4247c940045f90a4cc806e8b0b7dec101472c40262f357bc4fe87118ea18a74f not found: ID does not exist"
Sep 17 17:12:42.899996 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.899979    2577 scope.go:117] "RemoveContainer" containerID="18703cdb9e8fa31d03eaa888095c10cf344bcd367980961009d9baaa757481c4"
Sep 17 17:12:42.900195 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:12:42.900177    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18703cdb9e8fa31d03eaa888095c10cf344bcd367980961009d9baaa757481c4\": container with ID starting with 18703cdb9e8fa31d03eaa888095c10cf344bcd367980961009d9baaa757481c4 not found: ID does not exist" containerID="18703cdb9e8fa31d03eaa888095c10cf344bcd367980961009d9baaa757481c4"
Sep 17 17:12:42.900264 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.900201    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18703cdb9e8fa31d03eaa888095c10cf344bcd367980961009d9baaa757481c4"} err="failed to get container status \"18703cdb9e8fa31d03eaa888095c10cf344bcd367980961009d9baaa757481c4\": rpc error: code = NotFound desc = could not find container \"18703cdb9e8fa31d03eaa888095c10cf344bcd367980961009d9baaa757481c4\": container with ID starting with 18703cdb9e8fa31d03eaa888095c10cf344bcd367980961009d9baaa757481c4 not found: ID does not exist"
Sep 17 17:12:42.903796 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:42.903778    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-qhtb6"]
Sep 17 17:12:43.717836 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.717802    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"]
Sep 17 17:12:43.718212 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.718141    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kserve-container"
Sep 17 17:12:43.718212 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.718153    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kserve-container"
Sep 17 17:12:43.718212 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.718194    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="storage-initializer"
Sep 17 17:12:43.718212 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.718203    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="storage-initializer"
Sep 17 17:12:43.718357 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.718220    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kube-rbac-proxy"
Sep 17 17:12:43.718357 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.718227    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kube-rbac-proxy"
Sep 17 17:12:43.718357 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.718283    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kube-rbac-proxy"
Sep 17 17:12:43.718357 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.718296    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" containerName="kserve-container"
Sep 17 17:12:43.722853 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.722834    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:43.725257 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.725227    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 17:12:43.725380 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.725275    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 17:12:43.731496 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.731472    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"]
Sep 17 17:12:43.865483 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.865451    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1763b9ab-d209-46ff-9645-1aeb33b883f3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-7t86l\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:43.865483 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.865488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j2ns\" (UniqueName: \"kubernetes.io/projected/1763b9ab-d209-46ff-9645-1aeb33b883f3-kube-api-access-2j2ns\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-7t86l\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:43.865707 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.865521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1763b9ab-d209-46ff-9645-1aeb33b883f3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-7t86l\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:43.865707 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.865564    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1763b9ab-d209-46ff-9645-1aeb33b883f3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-7t86l\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:43.865707 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.865591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1763b9ab-d209-46ff-9645-1aeb33b883f3-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-7t86l\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:43.966572 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.966539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1763b9ab-d209-46ff-9645-1aeb33b883f3-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-7t86l\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:43.966759 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.966589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1763b9ab-d209-46ff-9645-1aeb33b883f3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-7t86l\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:43.966759 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.966618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j2ns\" (UniqueName: \"kubernetes.io/projected/1763b9ab-d209-46ff-9645-1aeb33b883f3-kube-api-access-2j2ns\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-7t86l\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:43.966759 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.966661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1763b9ab-d209-46ff-9645-1aeb33b883f3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-7t86l\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:43.966759 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.966692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1763b9ab-d209-46ff-9645-1aeb33b883f3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-7t86l\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:43.967179 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.967135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1763b9ab-d209-46ff-9645-1aeb33b883f3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-7t86l\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:43.967400 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.967382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1763b9ab-d209-46ff-9645-1aeb33b883f3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-7t86l\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:43.967438 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.967388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1763b9ab-d209-46ff-9645-1aeb33b883f3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-7t86l\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:43.968934 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.968884    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1763b9ab-d209-46ff-9645-1aeb33b883f3-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-7t86l\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:43.974582 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:43.974556    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j2ns\" (UniqueName: \"kubernetes.io/projected/1763b9ab-d209-46ff-9645-1aeb33b883f3-kube-api-access-2j2ns\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-7t86l\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:44.033623 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:44.033595    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:44.304019 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:44.303994    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"]
Sep 17 17:12:44.306254 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:12:44.306216    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1763b9ab_d209_46ff_9645_1aeb33b883f3.slice/crio-8affe4ccf24bab0aa50ee76409eab45b5a766f7aca3f04a2d430ec7d78ec1287 WatchSource:0}: Error finding container 8affe4ccf24bab0aa50ee76409eab45b5a766f7aca3f04a2d430ec7d78ec1287: Status 404 returned error can't find the container with id 8affe4ccf24bab0aa50ee76409eab45b5a766f7aca3f04a2d430ec7d78ec1287
Sep 17 17:12:44.452510 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:44.452479    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61678897-dc39-4999-ac2f-6d64111fe86d" path="/var/lib/kubelet/pods/61678897-dc39-4999-ac2f-6d64111fe86d/volumes"
Sep 17 17:12:44.886498 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:44.886461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" event={"ID":"1763b9ab-d209-46ff-9645-1aeb33b883f3","Type":"ContainerStarted","Data":"972b1492e4ec4c9881842ff450c888ef3e8f64c093d1c8df8834eb09e993ef3a"}
Sep 17 17:12:44.886866 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:44.886526    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" event={"ID":"1763b9ab-d209-46ff-9645-1aeb33b883f3","Type":"ContainerStarted","Data":"8affe4ccf24bab0aa50ee76409eab45b5a766f7aca3f04a2d430ec7d78ec1287"}
Sep 17 17:12:48.819380 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:48.819351    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"]
Sep 17 17:12:48.819832 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:48.819575    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kserve-container" containerID="cri-o://2f8a5e207d2af667b7885e52f95ae8cf5958c83935390a0a35555612036cd494" gracePeriod=30
Sep 17 17:12:48.819832 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:48.819620    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kube-rbac-proxy" containerID="cri-o://2017d58d3a7b53394f36932227ad2031c2ac02411f003b303b7e70a1e00bf736" gracePeriod=30
Sep 17 17:12:48.906838 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:48.906807    2577 generic.go:356] "Generic (PLEG): container finished" podID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerID="972b1492e4ec4c9881842ff450c888ef3e8f64c093d1c8df8834eb09e993ef3a" exitCode=0
Sep 17 17:12:48.906964 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:48.906847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" event={"ID":"1763b9ab-d209-46ff-9645-1aeb33b883f3","Type":"ContainerDied","Data":"972b1492e4ec4c9881842ff450c888ef3e8f64c093d1c8df8834eb09e993ef3a"}
Sep 17 17:12:49.913389 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:49.913353    2577 generic.go:356] "Generic (PLEG): container finished" podID="f975d9a2-3308-4893-80d3-27e16035fa56" containerID="2017d58d3a7b53394f36932227ad2031c2ac02411f003b303b7e70a1e00bf736" exitCode=2
Sep 17 17:12:49.913802 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:49.913432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" event={"ID":"f975d9a2-3308-4893-80d3-27e16035fa56","Type":"ContainerDied","Data":"2017d58d3a7b53394f36932227ad2031c2ac02411f003b303b7e70a1e00bf736"}
Sep 17 17:12:49.915671 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:49.915649    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" event={"ID":"1763b9ab-d209-46ff-9645-1aeb33b883f3","Type":"ContainerStarted","Data":"03e821b5407db1958dd57d113b5cffa69486cd14b21fdfbf5ce54704d27df216"}
Sep 17 17:12:49.915797 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:49.915696    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" event={"ID":"1763b9ab-d209-46ff-9645-1aeb33b883f3","Type":"ContainerStarted","Data":"b6f50b98da320bd1d2a4f749e97aebed6eb6d7509c3509174bb8b0ae81f21f36"}
Sep 17 17:12:49.915902 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:49.915886    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:49.935040 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:49.934999    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" podStartSLOduration=3.007820907 podStartE2EDuration="6.934986052s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.927165145s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:12:49.933226426 +0000 UTC m=+1887.958134714" watchObservedRunningTime="2026-09-17 17:12:49.934986052 +0000 UTC m=+1887.959894411"
Sep 17 17:12:50.920924 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:50.920893    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:50.922251 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:50.922224    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 17:12:51.520596 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:51.520551    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 17 17:12:51.525103 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:51.525068    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 17:12:51.925428 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:51.925379    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 17:12:53.664033 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.664006    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:12:53.746810 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.746783    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fzpg\" (UniqueName: \"kubernetes.io/projected/f975d9a2-3308-4893-80d3-27e16035fa56-kube-api-access-4fzpg\") pod \"f975d9a2-3308-4893-80d3-27e16035fa56\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") "
Sep 17 17:12:53.746978 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.746820    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f975d9a2-3308-4893-80d3-27e16035fa56-cabundle-cert\") pod \"f975d9a2-3308-4893-80d3-27e16035fa56\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") "
Sep 17 17:12:53.746978 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.746884    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f975d9a2-3308-4893-80d3-27e16035fa56-proxy-tls\") pod \"f975d9a2-3308-4893-80d3-27e16035fa56\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") "
Sep 17 17:12:53.746978 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.746906    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f975d9a2-3308-4893-80d3-27e16035fa56-kube-rbac-proxy-sar-config\") pod \"f975d9a2-3308-4893-80d3-27e16035fa56\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") "
Sep 17 17:12:53.746978 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.746933    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f975d9a2-3308-4893-80d3-27e16035fa56-kserve-provision-location\") pod \"f975d9a2-3308-4893-80d3-27e16035fa56\" (UID: \"f975d9a2-3308-4893-80d3-27e16035fa56\") "
Sep 17 17:12:53.747288 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.747258    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f975d9a2-3308-4893-80d3-27e16035fa56-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f975d9a2-3308-4893-80d3-27e16035fa56" (UID: "f975d9a2-3308-4893-80d3-27e16035fa56"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:12:53.747354 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.747286    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f975d9a2-3308-4893-80d3-27e16035fa56-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f975d9a2-3308-4893-80d3-27e16035fa56" (UID: "f975d9a2-3308-4893-80d3-27e16035fa56"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:12:53.747354 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.747298    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f975d9a2-3308-4893-80d3-27e16035fa56-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f975d9a2-3308-4893-80d3-27e16035fa56" (UID: "f975d9a2-3308-4893-80d3-27e16035fa56"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:12:53.748835 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.748813    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f975d9a2-3308-4893-80d3-27e16035fa56-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f975d9a2-3308-4893-80d3-27e16035fa56" (UID: "f975d9a2-3308-4893-80d3-27e16035fa56"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:12:53.748893 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.748861    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f975d9a2-3308-4893-80d3-27e16035fa56-kube-api-access-4fzpg" (OuterVolumeSpecName: "kube-api-access-4fzpg") pod "f975d9a2-3308-4893-80d3-27e16035fa56" (UID: "f975d9a2-3308-4893-80d3-27e16035fa56"). InnerVolumeSpecName "kube-api-access-4fzpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:12:53.848100 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.848068    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f975d9a2-3308-4893-80d3-27e16035fa56-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:12:53.848100 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.848096    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fzpg\" (UniqueName: \"kubernetes.io/projected/f975d9a2-3308-4893-80d3-27e16035fa56-kube-api-access-4fzpg\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:12:53.848100 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.848107    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f975d9a2-3308-4893-80d3-27e16035fa56-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:12:53.848359 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.848118    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f975d9a2-3308-4893-80d3-27e16035fa56-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:12:53.848359 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.848125    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f975d9a2-3308-4893-80d3-27e16035fa56-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:12:53.937526 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.937496    2577 generic.go:356] "Generic (PLEG): container finished" podID="f975d9a2-3308-4893-80d3-27e16035fa56" containerID="2f8a5e207d2af667b7885e52f95ae8cf5958c83935390a0a35555612036cd494" exitCode=0
Sep 17 17:12:53.937717 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.937547    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" event={"ID":"f975d9a2-3308-4893-80d3-27e16035fa56","Type":"ContainerDied","Data":"2f8a5e207d2af667b7885e52f95ae8cf5958c83935390a0a35555612036cd494"}
Sep 17 17:12:53.937717 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.937578    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"
Sep 17 17:12:53.937717 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.937607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b" event={"ID":"f975d9a2-3308-4893-80d3-27e16035fa56","Type":"ContainerDied","Data":"2ff92878aad28192276b274d8e17e21f86962356deeef470e4afd987d03589a1"}
Sep 17 17:12:53.937717 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.937644    2577 scope.go:117] "RemoveContainer" containerID="2017d58d3a7b53394f36932227ad2031c2ac02411f003b303b7e70a1e00bf736"
Sep 17 17:12:53.946864 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.946843    2577 scope.go:117] "RemoveContainer" containerID="2f8a5e207d2af667b7885e52f95ae8cf5958c83935390a0a35555612036cd494"
Sep 17 17:12:53.954542 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.954522    2577 scope.go:117] "RemoveContainer" containerID="a9dd2d3c7dda98a32b2850833eaf39b1c1ddfeb4c6b9c3b603595f0388e8f288"
Sep 17 17:12:53.962135 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.962104    2577 scope.go:117] "RemoveContainer" containerID="2017d58d3a7b53394f36932227ad2031c2ac02411f003b303b7e70a1e00bf736"
Sep 17 17:12:53.962426 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:12:53.962407    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2017d58d3a7b53394f36932227ad2031c2ac02411f003b303b7e70a1e00bf736\": container with ID starting with 2017d58d3a7b53394f36932227ad2031c2ac02411f003b303b7e70a1e00bf736 not found: ID does not exist" containerID="2017d58d3a7b53394f36932227ad2031c2ac02411f003b303b7e70a1e00bf736"
Sep 17 17:12:53.962507 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.962437    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2017d58d3a7b53394f36932227ad2031c2ac02411f003b303b7e70a1e00bf736"} err="failed to get container status \"2017d58d3a7b53394f36932227ad2031c2ac02411f003b303b7e70a1e00bf736\": rpc error: code = NotFound desc = could not find container \"2017d58d3a7b53394f36932227ad2031c2ac02411f003b303b7e70a1e00bf736\": container with ID starting with 2017d58d3a7b53394f36932227ad2031c2ac02411f003b303b7e70a1e00bf736 not found: ID does not exist"
Sep 17 17:12:53.962507 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.962454    2577 scope.go:117] "RemoveContainer" containerID="2f8a5e207d2af667b7885e52f95ae8cf5958c83935390a0a35555612036cd494"
Sep 17 17:12:53.962800 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:12:53.962770    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f8a5e207d2af667b7885e52f95ae8cf5958c83935390a0a35555612036cd494\": container with ID starting with 2f8a5e207d2af667b7885e52f95ae8cf5958c83935390a0a35555612036cd494 not found: ID does not exist" containerID="2f8a5e207d2af667b7885e52f95ae8cf5958c83935390a0a35555612036cd494"
Sep 17 17:12:53.962977 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.962807    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f8a5e207d2af667b7885e52f95ae8cf5958c83935390a0a35555612036cd494"} err="failed to get container status \"2f8a5e207d2af667b7885e52f95ae8cf5958c83935390a0a35555612036cd494\": rpc error: code = NotFound desc = could not find container \"2f8a5e207d2af667b7885e52f95ae8cf5958c83935390a0a35555612036cd494\": container with ID starting with 2f8a5e207d2af667b7885e52f95ae8cf5958c83935390a0a35555612036cd494 not found: ID does not exist"
Sep 17 17:12:53.962977 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.962831    2577 scope.go:117] "RemoveContainer" containerID="a9dd2d3c7dda98a32b2850833eaf39b1c1ddfeb4c6b9c3b603595f0388e8f288"
Sep 17 17:12:53.962977 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.962861    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"]
Sep 17 17:12:53.963209 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:12:53.963185    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9dd2d3c7dda98a32b2850833eaf39b1c1ddfeb4c6b9c3b603595f0388e8f288\": container with ID starting with a9dd2d3c7dda98a32b2850833eaf39b1c1ddfeb4c6b9c3b603595f0388e8f288 not found: ID does not exist" containerID="a9dd2d3c7dda98a32b2850833eaf39b1c1ddfeb4c6b9c3b603595f0388e8f288"
Sep 17 17:12:53.963278 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.963218    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9dd2d3c7dda98a32b2850833eaf39b1c1ddfeb4c6b9c3b603595f0388e8f288"} err="failed to get container status \"a9dd2d3c7dda98a32b2850833eaf39b1c1ddfeb4c6b9c3b603595f0388e8f288\": rpc error: code = NotFound desc = could not find container \"a9dd2d3c7dda98a32b2850833eaf39b1c1ddfeb4c6b9c3b603595f0388e8f288\": container with ID starting with a9dd2d3c7dda98a32b2850833eaf39b1c1ddfeb4c6b9c3b603595f0388e8f288 not found: ID does not exist"
Sep 17 17:12:53.964740 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:53.964723    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-58464fb4b5-bgs8b"]
Sep 17 17:12:54.007459 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.007432    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"]
Sep 17 17:12:54.007758 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.007745    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="storage-initializer"
Sep 17 17:12:54.007805 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.007761    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="storage-initializer"
Sep 17 17:12:54.007805 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.007776    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kserve-container"
Sep 17 17:12:54.007805 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.007782    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kserve-container"
Sep 17 17:12:54.007805 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.007796    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kube-rbac-proxy"
Sep 17 17:12:54.007805 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.007802    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kube-rbac-proxy"
Sep 17 17:12:54.007952 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.007854    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kube-rbac-proxy"
Sep 17 17:12:54.007952 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.007862    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" containerName="kserve-container"
Sep 17 17:12:54.012405 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.012390    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.015257 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.015238    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 17 17:12:54.015357 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.015252    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 17 17:12:54.022861 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.022840    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"]
Sep 17 17:12:54.150793 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.150704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnq2r\" (UniqueName: \"kubernetes.io/projected/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kube-api-access-bnq2r\") pod \"isvc-tensorflow-predictor-547dfc5669-4jrv2\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.150793 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.150744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-4jrv2\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.150793 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.150765    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-4jrv2\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.151019 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.150854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-4jrv2\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.151019 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.150898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-4jrv2\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.251960 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.251925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-4jrv2\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.252106 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.251972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-4jrv2\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.252106 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.252000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnq2r\" (UniqueName: \"kubernetes.io/projected/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kube-api-access-bnq2r\") pod \"isvc-tensorflow-predictor-547dfc5669-4jrv2\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.252106 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.252020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-4jrv2\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.252106 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.252037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-4jrv2\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.252361 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.252342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-4jrv2\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.252835 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.252814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-4jrv2\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.252876 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.252814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-4jrv2\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.254429 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.254410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-4jrv2\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.261239 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.261219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnq2r\" (UniqueName: \"kubernetes.io/projected/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kube-api-access-bnq2r\") pod \"isvc-tensorflow-predictor-547dfc5669-4jrv2\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.323027 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.322995    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:12:54.448114 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.448039    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f975d9a2-3308-4893-80d3-27e16035fa56" path="/var/lib/kubelet/pods/f975d9a2-3308-4893-80d3-27e16035fa56/volumes"
Sep 17 17:12:54.598953 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.598927    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"]
Sep 17 17:12:54.601253 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:12:54.601208    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc71810_bc2f_41c6_afe2_c9dec5e7a832.slice/crio-1d01e46645efb9f343edda8c625ece1dd6b3e41fc3f89502198a4c9d2e7b4276 WatchSource:0}: Error finding container 1d01e46645efb9f343edda8c625ece1dd6b3e41fc3f89502198a4c9d2e7b4276: Status 404 returned error can't find the container with id 1d01e46645efb9f343edda8c625ece1dd6b3e41fc3f89502198a4c9d2e7b4276
Sep 17 17:12:54.947470 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.947426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" event={"ID":"5bc71810-bc2f-41c6-afe2-c9dec5e7a832","Type":"ContainerStarted","Data":"9c58119d874e03968997e911ffdb56c431a65edd70d27bf98f25cefa420498e6"}
Sep 17 17:12:54.947962 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:54.947497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" event={"ID":"5bc71810-bc2f-41c6-afe2-c9dec5e7a832","Type":"ContainerStarted","Data":"1d01e46645efb9f343edda8c625ece1dd6b3e41fc3f89502198a4c9d2e7b4276"}
Sep 17 17:12:56.929619 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:56.929589    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:12:56.930262 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:56.930233    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 17:12:59.973107 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:59.973074    2577 generic.go:356] "Generic (PLEG): container finished" podID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerID="9c58119d874e03968997e911ffdb56c431a65edd70d27bf98f25cefa420498e6" exitCode=0
Sep 17 17:12:59.973554 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:59.973138    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" event={"ID":"5bc71810-bc2f-41c6-afe2-c9dec5e7a832","Type":"ContainerDied","Data":"9c58119d874e03968997e911ffdb56c431a65edd70d27bf98f25cefa420498e6"}
Sep 17 17:12:59.974249 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:12:59.974233    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:13:03.996125 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:03.996096    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" event={"ID":"5bc71810-bc2f-41c6-afe2-c9dec5e7a832","Type":"ContainerStarted","Data":"9c51fa7bbb99ef99ddfb3c81a2b13324c8904f5ae4f5e85631f1cad323921a41"}
Sep 17 17:13:03.996509 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:03.996192    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" event={"ID":"5bc71810-bc2f-41c6-afe2-c9dec5e7a832","Type":"ContainerStarted","Data":"81c2fbf99bf15e431c88415f8fb5abf64e27a8e3624c58095112c5667006c1da"}
Sep 17 17:13:03.996509 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:03.996415    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:13:04.018333 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:04.018289    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" podStartSLOduration=2.8033193929999998 podStartE2EDuration="11.018275871s" totalImagesPullingTime="3.862479999s" totalInitContainerRuntime="4.352476479s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:13:04.016988064 +0000 UTC m=+1902.041896364" watchObservedRunningTime="2026-09-17 17:13:04.018275871 +0000 UTC m=+1902.043184159"
Sep 17 17:13:05.001179 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:05.001132    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:13:05.002227 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:05.002198    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 17:13:06.005621 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:06.005569    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 17:13:06.930134 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:06.930098    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 17:13:11.010799 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:11.010763    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:13:11.011469 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:11.011429    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 17:13:16.930701 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:16.930662    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 17:13:21.012508 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:21.012480    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:13:26.930448 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:26.930403    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 17:13:34.011415 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:34.011374    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"]
Sep 17 17:13:34.011836 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:34.011712    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kserve-container" containerID="cri-o://81c2fbf99bf15e431c88415f8fb5abf64e27a8e3624c58095112c5667006c1da" gracePeriod=30
Sep 17 17:13:34.012096 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:34.011803    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kube-rbac-proxy" containerID="cri-o://9c51fa7bbb99ef99ddfb3c81a2b13324c8904f5ae4f5e85631f1cad323921a41" gracePeriod=30
Sep 17 17:13:34.138462 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:34.138436    2577 generic.go:356] "Generic (PLEG): container finished" podID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerID="9c51fa7bbb99ef99ddfb3c81a2b13324c8904f5ae4f5e85631f1cad323921a41" exitCode=2
Sep 17 17:13:34.138584 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:34.138496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" event={"ID":"5bc71810-bc2f-41c6-afe2-c9dec5e7a832","Type":"ContainerDied","Data":"9c51fa7bbb99ef99ddfb3c81a2b13324c8904f5ae4f5e85631f1cad323921a41"}
Sep 17 17:13:36.005899 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:36.005853    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 17 17:13:36.930173 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:36.930124    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 17:13:39.210365 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.210333    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"]
Sep 17 17:13:39.213921 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.213902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.216345 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.216321    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 17 17:13:39.216442 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.216364    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 17 17:13:39.222871 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.222851    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"]
Sep 17 17:13:39.293055 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.293023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d5cd\" (UniqueName: \"kubernetes.io/projected/d16d3aa2-07ae-4ae8-a656-1471c7903458-kube-api-access-7d5cd\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.293203 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.293067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d16d3aa2-07ae-4ae8-a656-1471c7903458-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.293282 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.293213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d16d3aa2-07ae-4ae8-a656-1471c7903458-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.293282 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.293259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d16d3aa2-07ae-4ae8-a656-1471c7903458-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.293387 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.293288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d16d3aa2-07ae-4ae8-a656-1471c7903458-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.394033 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.394010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d16d3aa2-07ae-4ae8-a656-1471c7903458-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.394135 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.394039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d16d3aa2-07ae-4ae8-a656-1471c7903458-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.394135 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.394056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d16d3aa2-07ae-4ae8-a656-1471c7903458-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.394135 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.394082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d5cd\" (UniqueName: \"kubernetes.io/projected/d16d3aa2-07ae-4ae8-a656-1471c7903458-kube-api-access-7d5cd\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.394135 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.394110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d16d3aa2-07ae-4ae8-a656-1471c7903458-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.394398 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:13:39.394213    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 17 17:13:39.394398 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:13:39.394277    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d16d3aa2-07ae-4ae8-a656-1471c7903458-proxy-tls podName:d16d3aa2-07ae-4ae8-a656-1471c7903458 nodeName:}" failed. No retries permitted until 2026-09-17 17:13:39.89425768 +0000 UTC m=+1937.919165956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d16d3aa2-07ae-4ae8-a656-1471c7903458-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" (UID: "d16d3aa2-07ae-4ae8-a656-1471c7903458") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 17 17:13:39.394570 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.394551    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d16d3aa2-07ae-4ae8-a656-1471c7903458-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.394777 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.394758    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d16d3aa2-07ae-4ae8-a656-1471c7903458-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.394849 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.394779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d16d3aa2-07ae-4ae8-a656-1471c7903458-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.404541 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.404518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d5cd\" (UniqueName: \"kubernetes.io/projected/d16d3aa2-07ae-4ae8-a656-1471c7903458-kube-api-access-7d5cd\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.898459 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.898426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d16d3aa2-07ae-4ae8-a656-1471c7903458-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:39.900854 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:39.900819    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d16d3aa2-07ae-4ae8-a656-1471c7903458-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:40.124321 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:40.124288    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:40.395351 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:40.395331    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"]
Sep 17 17:13:40.398144 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:13:40.398117    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16d3aa2_07ae_4ae8_a656_1471c7903458.slice/crio-b3a59f68d08812efdbd1942a9458a4604f645e089b546e29ba0a117c46cf6bd5 WatchSource:0}: Error finding container b3a59f68d08812efdbd1942a9458a4604f645e089b546e29ba0a117c46cf6bd5: Status 404 returned error can't find the container with id b3a59f68d08812efdbd1942a9458a4604f645e089b546e29ba0a117c46cf6bd5
Sep 17 17:13:41.006766 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:41.006722    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 17 17:13:41.170947 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:41.170911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" event={"ID":"d16d3aa2-07ae-4ae8-a656-1471c7903458","Type":"ContainerStarted","Data":"ad8106273412f410528cb30d4bd8e7ebcba12136f119fef4e9ff4660a0abd6a3"}
Sep 17 17:13:41.171199 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:41.170967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" event={"ID":"d16d3aa2-07ae-4ae8-a656-1471c7903458","Type":"ContainerStarted","Data":"b3a59f68d08812efdbd1942a9458a4604f645e089b546e29ba0a117c46cf6bd5"}
Sep 17 17:13:46.006212 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:46.006140    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 17 17:13:46.006684 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:46.006278    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:13:46.195694 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:46.195657    2577 generic.go:356] "Generic (PLEG): container finished" podID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerID="ad8106273412f410528cb30d4bd8e7ebcba12136f119fef4e9ff4660a0abd6a3" exitCode=0
Sep 17 17:13:46.195876 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:46.195710    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" event={"ID":"d16d3aa2-07ae-4ae8-a656-1471c7903458","Type":"ContainerDied","Data":"ad8106273412f410528cb30d4bd8e7ebcba12136f119fef4e9ff4660a0abd6a3"}
Sep 17 17:13:46.930997 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:46.930961    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 17:13:47.203216 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:47.203120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" event={"ID":"d16d3aa2-07ae-4ae8-a656-1471c7903458","Type":"ContainerStarted","Data":"54b211bc36ed47f2386115f1596b37cb4e248429993edcaaeb0e676952170cf1"}
Sep 17 17:13:47.203216 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:47.203200    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" event={"ID":"d16d3aa2-07ae-4ae8-a656-1471c7903458","Type":"ContainerStarted","Data":"4d7c0e6b8db0e349a81a11fdebb4023974e5504d599a565f9d8feb0a03272748"}
Sep 17 17:13:47.203677 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:47.203453    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:47.203677 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:47.203611    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:47.204865 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:47.204835    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 17:13:47.222991 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:47.222943    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" podStartSLOduration=3.497290408 podStartE2EDuration="8.222929154s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.725638746s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:13:47.222076763 +0000 UTC m=+1945.246985051" watchObservedRunningTime="2026-09-17 17:13:47.222929154 +0000 UTC m=+1945.247837442"
Sep 17 17:13:48.209107 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:48.209061    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 17:13:51.006530 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:51.006468    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 17 17:13:53.214093 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:53.214064    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:13:53.214785 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:53.214755    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 17:13:56.005924 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:56.005878    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 17 17:13:56.930106 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:13:56.930068    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 17:14:01.006584 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:01.006535    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 17 17:14:03.214877 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:03.214849    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:14:04.287140 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.287070    2577 generic.go:356] "Generic (PLEG): container finished" podID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerID="81c2fbf99bf15e431c88415f8fb5abf64e27a8e3624c58095112c5667006c1da" exitCode=137
Sep 17 17:14:04.287481 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.287139    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" event={"ID":"5bc71810-bc2f-41c6-afe2-c9dec5e7a832","Type":"ContainerDied","Data":"81c2fbf99bf15e431c88415f8fb5abf64e27a8e3624c58095112c5667006c1da"}
Sep 17 17:14:04.649305 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.649281    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:14:04.798679 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.798645    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kserve-provision-location\") pod \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") "
Sep 17 17:14:04.798841 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.798713    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-cabundle-cert\") pod \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") "
Sep 17 17:14:04.798841 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.798739    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-proxy-tls\") pod \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") "
Sep 17 17:14:04.798841 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.798790    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kube-rbac-proxy-sar-config\") pod \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") "
Sep 17 17:14:04.798995 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.798924    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnq2r\" (UniqueName: \"kubernetes.io/projected/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kube-api-access-bnq2r\") pod \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\" (UID: \"5bc71810-bc2f-41c6-afe2-c9dec5e7a832\") "
Sep 17 17:14:04.799083 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.799061    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5bc71810-bc2f-41c6-afe2-c9dec5e7a832" (UID: "5bc71810-bc2f-41c6-afe2-c9dec5e7a832"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:14:04.799083 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.799074    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5bc71810-bc2f-41c6-afe2-c9dec5e7a832" (UID: "5bc71810-bc2f-41c6-afe2-c9dec5e7a832"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:14:04.799226 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.799204    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:14:04.799226 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.799216    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:14:04.800813 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.800790    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5bc71810-bc2f-41c6-afe2-c9dec5e7a832" (UID: "5bc71810-bc2f-41c6-afe2-c9dec5e7a832"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:14:04.800990 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.800967    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kube-api-access-bnq2r" (OuterVolumeSpecName: "kube-api-access-bnq2r") pod "5bc71810-bc2f-41c6-afe2-c9dec5e7a832" (UID: "5bc71810-bc2f-41c6-afe2-c9dec5e7a832"). InnerVolumeSpecName "kube-api-access-bnq2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:14:04.809588 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.809564    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5bc71810-bc2f-41c6-afe2-c9dec5e7a832" (UID: "5bc71810-bc2f-41c6-afe2-c9dec5e7a832"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:04.899779 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.899716    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnq2r\" (UniqueName: \"kubernetes.io/projected/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kube-api-access-bnq2r\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:14:04.899779 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.899746    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:14:04.899779 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:04.899757    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bc71810-bc2f-41c6-afe2-c9dec5e7a832-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:14:05.294123 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:05.294100    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"
Sep 17 17:14:05.294584 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:05.294120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2" event={"ID":"5bc71810-bc2f-41c6-afe2-c9dec5e7a832","Type":"ContainerDied","Data":"1d01e46645efb9f343edda8c625ece1dd6b3e41fc3f89502198a4c9d2e7b4276"}
Sep 17 17:14:05.294584 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:05.294200    2577 scope.go:117] "RemoveContainer" containerID="9c51fa7bbb99ef99ddfb3c81a2b13324c8904f5ae4f5e85631f1cad323921a41"
Sep 17 17:14:05.303698 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:05.303681    2577 scope.go:117] "RemoveContainer" containerID="81c2fbf99bf15e431c88415f8fb5abf64e27a8e3624c58095112c5667006c1da"
Sep 17 17:14:05.311035 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:05.311019    2577 scope.go:117] "RemoveContainer" containerID="9c58119d874e03968997e911ffdb56c431a65edd70d27bf98f25cefa420498e6"
Sep 17 17:14:05.316473 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:05.316453    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"]
Sep 17 17:14:05.320314 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:05.320294    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-4jrv2"]
Sep 17 17:14:06.110272 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:06.110236    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"]
Sep 17 17:14:06.110572 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:06.110543    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kserve-container" containerID="cri-o://b6f50b98da320bd1d2a4f749e97aebed6eb6d7509c3509174bb8b0ae81f21f36" gracePeriod=30
Sep 17 17:14:06.110657 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:06.110588    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kube-rbac-proxy" containerID="cri-o://03e821b5407db1958dd57d113b5cffa69486cd14b21fdfbf5ce54704d27df216" gracePeriod=30
Sep 17 17:14:06.111373 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:06.111355    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:14:06.301736 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:06.301711    2577 generic.go:356] "Generic (PLEG): container finished" podID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerID="03e821b5407db1958dd57d113b5cffa69486cd14b21fdfbf5ce54704d27df216" exitCode=2
Sep 17 17:14:06.302062 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:06.301785    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" event={"ID":"1763b9ab-d209-46ff-9645-1aeb33b883f3","Type":"ContainerDied","Data":"03e821b5407db1958dd57d113b5cffa69486cd14b21fdfbf5ce54704d27df216"}
Sep 17 17:14:06.446651 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:06.446594    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" path="/var/lib/kubelet/pods/5bc71810-bc2f-41c6-afe2-c9dec5e7a832/volumes"
Sep 17 17:14:06.926229 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:06.926188    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 17 17:14:10.160188 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.160148    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:14:10.324417 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.324387    2577 generic.go:356] "Generic (PLEG): container finished" podID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerID="b6f50b98da320bd1d2a4f749e97aebed6eb6d7509c3509174bb8b0ae81f21f36" exitCode=0
Sep 17 17:14:10.324588 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.324440    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"
Sep 17 17:14:10.324588 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.324475    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" event={"ID":"1763b9ab-d209-46ff-9645-1aeb33b883f3","Type":"ContainerDied","Data":"b6f50b98da320bd1d2a4f749e97aebed6eb6d7509c3509174bb8b0ae81f21f36"}
Sep 17 17:14:10.324588 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.324540    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l" event={"ID":"1763b9ab-d209-46ff-9645-1aeb33b883f3","Type":"ContainerDied","Data":"8affe4ccf24bab0aa50ee76409eab45b5a766f7aca3f04a2d430ec7d78ec1287"}
Sep 17 17:14:10.324588 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.324568    2577 scope.go:117] "RemoveContainer" containerID="03e821b5407db1958dd57d113b5cffa69486cd14b21fdfbf5ce54704d27df216"
Sep 17 17:14:10.332604 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.332589    2577 scope.go:117] "RemoveContainer" containerID="b6f50b98da320bd1d2a4f749e97aebed6eb6d7509c3509174bb8b0ae81f21f36"
Sep 17 17:14:10.338646 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.338628    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1763b9ab-d209-46ff-9645-1aeb33b883f3-proxy-tls\") pod \"1763b9ab-d209-46ff-9645-1aeb33b883f3\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") "
Sep 17 17:14:10.338724 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.338664    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1763b9ab-d209-46ff-9645-1aeb33b883f3-cabundle-cert\") pod \"1763b9ab-d209-46ff-9645-1aeb33b883f3\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") "
Sep 17 17:14:10.338724 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.338692    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1763b9ab-d209-46ff-9645-1aeb33b883f3-kserve-provision-location\") pod \"1763b9ab-d209-46ff-9645-1aeb33b883f3\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") "
Sep 17 17:14:10.338831 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.338812    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1763b9ab-d209-46ff-9645-1aeb33b883f3-kube-rbac-proxy-sar-config\") pod \"1763b9ab-d209-46ff-9645-1aeb33b883f3\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") "
Sep 17 17:14:10.339100 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.338850    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j2ns\" (UniqueName: \"kubernetes.io/projected/1763b9ab-d209-46ff-9645-1aeb33b883f3-kube-api-access-2j2ns\") pod \"1763b9ab-d209-46ff-9645-1aeb33b883f3\" (UID: \"1763b9ab-d209-46ff-9645-1aeb33b883f3\") "
Sep 17 17:14:10.339100 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.339014    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1763b9ab-d209-46ff-9645-1aeb33b883f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1763b9ab-d209-46ff-9645-1aeb33b883f3" (UID: "1763b9ab-d209-46ff-9645-1aeb33b883f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:10.339275 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.339113    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1763b9ab-d209-46ff-9645-1aeb33b883f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1763b9ab-d209-46ff-9645-1aeb33b883f3" (UID: "1763b9ab-d209-46ff-9645-1aeb33b883f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:14:10.339275 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.339215    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1763b9ab-d209-46ff-9645-1aeb33b883f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1763b9ab-d209-46ff-9645-1aeb33b883f3" (UID: "1763b9ab-d209-46ff-9645-1aeb33b883f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:14:10.339359 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.339298    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1763b9ab-d209-46ff-9645-1aeb33b883f3-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:14:10.339359 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.339320    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1763b9ab-d209-46ff-9645-1aeb33b883f3-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:14:10.339359 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.339335    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1763b9ab-d209-46ff-9645-1aeb33b883f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:14:10.339878 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.339857    2577 scope.go:117] "RemoveContainer" containerID="972b1492e4ec4c9881842ff450c888ef3e8f64c093d1c8df8834eb09e993ef3a"
Sep 17 17:14:10.340864 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.340841    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1763b9ab-d209-46ff-9645-1aeb33b883f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1763b9ab-d209-46ff-9645-1aeb33b883f3" (UID: "1763b9ab-d209-46ff-9645-1aeb33b883f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:14:10.341041 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.340970    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1763b9ab-d209-46ff-9645-1aeb33b883f3-kube-api-access-2j2ns" (OuterVolumeSpecName: "kube-api-access-2j2ns") pod "1763b9ab-d209-46ff-9645-1aeb33b883f3" (UID: "1763b9ab-d209-46ff-9645-1aeb33b883f3"). InnerVolumeSpecName "kube-api-access-2j2ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:14:10.354218 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.354153    2577 scope.go:117] "RemoveContainer" containerID="03e821b5407db1958dd57d113b5cffa69486cd14b21fdfbf5ce54704d27df216"
Sep 17 17:14:10.354502 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:14:10.354478    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e821b5407db1958dd57d113b5cffa69486cd14b21fdfbf5ce54704d27df216\": container with ID starting with 03e821b5407db1958dd57d113b5cffa69486cd14b21fdfbf5ce54704d27df216 not found: ID does not exist" containerID="03e821b5407db1958dd57d113b5cffa69486cd14b21fdfbf5ce54704d27df216"
Sep 17 17:14:10.354557 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.354510    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e821b5407db1958dd57d113b5cffa69486cd14b21fdfbf5ce54704d27df216"} err="failed to get container status \"03e821b5407db1958dd57d113b5cffa69486cd14b21fdfbf5ce54704d27df216\": rpc error: code = NotFound desc = could not find container \"03e821b5407db1958dd57d113b5cffa69486cd14b21fdfbf5ce54704d27df216\": container with ID starting with 03e821b5407db1958dd57d113b5cffa69486cd14b21fdfbf5ce54704d27df216 not found: ID does not exist"
Sep 17 17:14:10.354557 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.354526    2577 scope.go:117] "RemoveContainer" containerID="b6f50b98da320bd1d2a4f749e97aebed6eb6d7509c3509174bb8b0ae81f21f36"
Sep 17 17:14:10.354777 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:14:10.354758    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f50b98da320bd1d2a4f749e97aebed6eb6d7509c3509174bb8b0ae81f21f36\": container with ID starting with b6f50b98da320bd1d2a4f749e97aebed6eb6d7509c3509174bb8b0ae81f21f36 not found: ID does not exist" containerID="b6f50b98da320bd1d2a4f749e97aebed6eb6d7509c3509174bb8b0ae81f21f36"
Sep 17 17:14:10.354824 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.354780    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f50b98da320bd1d2a4f749e97aebed6eb6d7509c3509174bb8b0ae81f21f36"} err="failed to get container status \"b6f50b98da320bd1d2a4f749e97aebed6eb6d7509c3509174bb8b0ae81f21f36\": rpc error: code = NotFound desc = could not find container \"b6f50b98da320bd1d2a4f749e97aebed6eb6d7509c3509174bb8b0ae81f21f36\": container with ID starting with b6f50b98da320bd1d2a4f749e97aebed6eb6d7509c3509174bb8b0ae81f21f36 not found: ID does not exist"
Sep 17 17:14:10.354824 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.354797    2577 scope.go:117] "RemoveContainer" containerID="972b1492e4ec4c9881842ff450c888ef3e8f64c093d1c8df8834eb09e993ef3a"
Sep 17 17:14:10.355023 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:14:10.355008    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"972b1492e4ec4c9881842ff450c888ef3e8f64c093d1c8df8834eb09e993ef3a\": container with ID starting with 972b1492e4ec4c9881842ff450c888ef3e8f64c093d1c8df8834eb09e993ef3a not found: ID does not exist" containerID="972b1492e4ec4c9881842ff450c888ef3e8f64c093d1c8df8834eb09e993ef3a"
Sep 17 17:14:10.355059 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.355026    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"972b1492e4ec4c9881842ff450c888ef3e8f64c093d1c8df8834eb09e993ef3a"} err="failed to get container status \"972b1492e4ec4c9881842ff450c888ef3e8f64c093d1c8df8834eb09e993ef3a\": rpc error: code = NotFound desc = could not find container \"972b1492e4ec4c9881842ff450c888ef3e8f64c093d1c8df8834eb09e993ef3a\": container with ID starting with 972b1492e4ec4c9881842ff450c888ef3e8f64c093d1c8df8834eb09e993ef3a not found: ID does not exist"
Sep 17 17:14:10.440000 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.439980    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1763b9ab-d209-46ff-9645-1aeb33b883f3-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:14:10.440000 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.439998    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j2ns\" (UniqueName: \"kubernetes.io/projected/1763b9ab-d209-46ff-9645-1aeb33b883f3-kube-api-access-2j2ns\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:14:10.642613 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.642547    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"]
Sep 17 17:14:10.647800 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:10.647779    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-7t86l"]
Sep 17 17:14:12.446432 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:12.446402    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" path="/var/lib/kubelet/pods/1763b9ab-d209-46ff-9645-1aeb33b883f3/volumes"
Sep 17 17:14:13.702714 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.702683    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"]
Sep 17 17:14:13.703054 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.702981    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="storage-initializer"
Sep 17 17:14:13.703054 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.702992    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="storage-initializer"
Sep 17 17:14:13.703054 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.703001    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kube-rbac-proxy"
Sep 17 17:14:13.703054 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.703006    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kube-rbac-proxy"
Sep 17 17:14:13.703054 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.703013    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="storage-initializer"
Sep 17 17:14:13.703054 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.703019    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="storage-initializer"
Sep 17 17:14:13.703054 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.703028    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kserve-container"
Sep 17 17:14:13.703054 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.703034    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kserve-container"
Sep 17 17:14:13.703054 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.703049    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kserve-container"
Sep 17 17:14:13.703054 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.703055    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kserve-container"
Sep 17 17:14:13.703376 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.703061    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kube-rbac-proxy"
Sep 17 17:14:13.703376 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.703066    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kube-rbac-proxy"
Sep 17 17:14:13.703376 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.703124    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kube-rbac-proxy"
Sep 17 17:14:13.703376 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.703129    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1763b9ab-d209-46ff-9645-1aeb33b883f3" containerName="kserve-container"
Sep 17 17:14:13.703376 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.703137    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kube-rbac-proxy"
Sep 17 17:14:13.703376 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.703142    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bc71810-bc2f-41c6-afe2-c9dec5e7a832" containerName="kserve-container"
Sep 17 17:14:13.707546 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.707527    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:13.709947 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.709926    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 17:14:13.710052 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.709929    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 17:14:13.716008 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.715983    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"]
Sep 17 17:14:13.862900 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.862873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-wkr97\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:13.863053 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.862911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-wkr97\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:13.863053 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.862934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-wkr97\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:13.863053 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.863024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4fxn\" (UniqueName: \"kubernetes.io/projected/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kube-api-access-l4fxn\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-wkr97\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:13.863183 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.863071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-wkr97\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:13.964259 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.964203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-wkr97\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:13.964259 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.964237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-wkr97\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:13.964259 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.964259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-wkr97\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:13.964446 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.964278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4fxn\" (UniqueName: \"kubernetes.io/projected/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kube-api-access-l4fxn\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-wkr97\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:13.964446 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.964303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-wkr97\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:13.964681 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.964661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-wkr97\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:13.964941 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.964924    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-wkr97\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:13.964987 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.964977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-wkr97\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:13.966589 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.966566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-wkr97\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:13.972070 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:13.972049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4fxn\" (UniqueName: \"kubernetes.io/projected/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kube-api-access-l4fxn\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-wkr97\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:14.017557 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:14.017535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:14.288941 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:14.288920    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"]
Sep 17 17:14:14.290942 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:14:14.290914    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ddac1ea_192f_4b56_ab02_65f0fa3c6867.slice/crio-a5e13752857006424d42f5a52359bc5ddf9c7494933422587a9105d62c973f42 WatchSource:0}: Error finding container a5e13752857006424d42f5a52359bc5ddf9c7494933422587a9105d62c973f42: Status 404 returned error can't find the container with id a5e13752857006424d42f5a52359bc5ddf9c7494933422587a9105d62c973f42
Sep 17 17:14:14.343283 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:14.343257    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" event={"ID":"9ddac1ea-192f-4b56-ab02-65f0fa3c6867","Type":"ContainerStarted","Data":"a5e13752857006424d42f5a52359bc5ddf9c7494933422587a9105d62c973f42"}
Sep 17 17:14:15.349052 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:15.348964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" event={"ID":"9ddac1ea-192f-4b56-ab02-65f0fa3c6867","Type":"ContainerStarted","Data":"794ba5e5b187883801f37083da2b55eadba62ac7a556c3f18d943250e612d59b"}
Sep 17 17:14:18.364110 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:18.364076    2577 generic.go:356] "Generic (PLEG): container finished" podID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerID="794ba5e5b187883801f37083da2b55eadba62ac7a556c3f18d943250e612d59b" exitCode=0
Sep 17 17:14:18.364430 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:18.364146    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" event={"ID":"9ddac1ea-192f-4b56-ab02-65f0fa3c6867","Type":"ContainerDied","Data":"794ba5e5b187883801f37083da2b55eadba62ac7a556c3f18d943250e612d59b"}
Sep 17 17:14:19.117109 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:19.117076    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"]
Sep 17 17:14:19.117350 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:19.117326    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kserve-container" containerID="cri-o://4d7c0e6b8db0e349a81a11fdebb4023974e5504d599a565f9d8feb0a03272748" gracePeriod=30
Sep 17 17:14:19.117438 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:19.117354    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kube-rbac-proxy" containerID="cri-o://54b211bc36ed47f2386115f1596b37cb4e248429993edcaaeb0e676952170cf1" gracePeriod=30
Sep 17 17:14:19.370406 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:19.370334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" event={"ID":"9ddac1ea-192f-4b56-ab02-65f0fa3c6867","Type":"ContainerStarted","Data":"54b1b8b3f520253158bda0560137e72abeddebde0154aa8a2926c1e8247a7375"}
Sep 17 17:14:19.370780 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:19.370408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" event={"ID":"9ddac1ea-192f-4b56-ab02-65f0fa3c6867","Type":"ContainerStarted","Data":"ae2cf0e3670ab2d7a11ab4a486219915cb4316d5ee131335beb25c08d7888819"}
Sep 17 17:14:19.370780 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:19.370621    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:19.372541 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:19.372517    2577 generic.go:356] "Generic (PLEG): container finished" podID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerID="54b211bc36ed47f2386115f1596b37cb4e248429993edcaaeb0e676952170cf1" exitCode=2
Sep 17 17:14:19.372635 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:19.372582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" event={"ID":"d16d3aa2-07ae-4ae8-a656-1471c7903458","Type":"ContainerDied","Data":"54b211bc36ed47f2386115f1596b37cb4e248429993edcaaeb0e676952170cf1"}
Sep 17 17:14:19.391171 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:19.391118    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" podStartSLOduration=2.542043933 podStartE2EDuration="6.391105849s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.849061916s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:14:19.389804848 +0000 UTC m=+1977.414713159" watchObservedRunningTime="2026-09-17 17:14:19.391105849 +0000 UTC m=+1977.416014136"
Sep 17 17:14:20.376934 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:20.376903    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:20.378310 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:20.378279    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 17:14:21.381108 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:21.381065    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 17:14:23.209870 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:23.209832    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 17:14:26.386089 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:26.386056    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:14:26.386631 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:26.386602    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 17:14:28.209432 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:28.209391    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 17:14:33.209978 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:33.209940    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 17:14:33.210442 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:33.210077    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:14:36.387506 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:36.387461    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 17:14:38.209521 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:38.209482    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 17:14:43.209654 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:43.209605    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 17:14:46.387424 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:46.387377    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 17:14:48.209378 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:48.209338    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 17:14:49.512951 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.512871    2577 generic.go:356] "Generic (PLEG): container finished" podID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerID="4d7c0e6b8db0e349a81a11fdebb4023974e5504d599a565f9d8feb0a03272748" exitCode=137
Sep 17 17:14:49.513295 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.512942    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" event={"ID":"d16d3aa2-07ae-4ae8-a656-1471c7903458","Type":"ContainerDied","Data":"4d7c0e6b8db0e349a81a11fdebb4023974e5504d599a565f9d8feb0a03272748"}
Sep 17 17:14:49.762805 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.762779    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:14:49.825984 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.825953    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d16d3aa2-07ae-4ae8-a656-1471c7903458-proxy-tls\") pod \"d16d3aa2-07ae-4ae8-a656-1471c7903458\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") "
Sep 17 17:14:49.826130 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.826020    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d16d3aa2-07ae-4ae8-a656-1471c7903458-kserve-provision-location\") pod \"d16d3aa2-07ae-4ae8-a656-1471c7903458\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") "
Sep 17 17:14:49.826130 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.826052    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d5cd\" (UniqueName: \"kubernetes.io/projected/d16d3aa2-07ae-4ae8-a656-1471c7903458-kube-api-access-7d5cd\") pod \"d16d3aa2-07ae-4ae8-a656-1471c7903458\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") "
Sep 17 17:14:49.826130 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.826081    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d16d3aa2-07ae-4ae8-a656-1471c7903458-kube-rbac-proxy-sar-config\") pod \"d16d3aa2-07ae-4ae8-a656-1471c7903458\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") "
Sep 17 17:14:49.826277 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.826139    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d16d3aa2-07ae-4ae8-a656-1471c7903458-cabundle-cert\") pod \"d16d3aa2-07ae-4ae8-a656-1471c7903458\" (UID: \"d16d3aa2-07ae-4ae8-a656-1471c7903458\") "
Sep 17 17:14:49.826522 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.826486    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16d3aa2-07ae-4ae8-a656-1471c7903458-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d16d3aa2-07ae-4ae8-a656-1471c7903458" (UID: "d16d3aa2-07ae-4ae8-a656-1471c7903458"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:14:49.826607 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.826543    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16d3aa2-07ae-4ae8-a656-1471c7903458-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d16d3aa2-07ae-4ae8-a656-1471c7903458" (UID: "d16d3aa2-07ae-4ae8-a656-1471c7903458"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:14:49.828104 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.828081    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16d3aa2-07ae-4ae8-a656-1471c7903458-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d16d3aa2-07ae-4ae8-a656-1471c7903458" (UID: "d16d3aa2-07ae-4ae8-a656-1471c7903458"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:14:49.828220 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.828117    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16d3aa2-07ae-4ae8-a656-1471c7903458-kube-api-access-7d5cd" (OuterVolumeSpecName: "kube-api-access-7d5cd") pod "d16d3aa2-07ae-4ae8-a656-1471c7903458" (UID: "d16d3aa2-07ae-4ae8-a656-1471c7903458"). InnerVolumeSpecName "kube-api-access-7d5cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:14:49.834476 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.834453    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16d3aa2-07ae-4ae8-a656-1471c7903458-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d16d3aa2-07ae-4ae8-a656-1471c7903458" (UID: "d16d3aa2-07ae-4ae8-a656-1471c7903458"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:49.927098 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.927066    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d16d3aa2-07ae-4ae8-a656-1471c7903458-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:14:49.927098 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.927091    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d16d3aa2-07ae-4ae8-a656-1471c7903458-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:14:49.927098 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.927102    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d16d3aa2-07ae-4ae8-a656-1471c7903458-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:14:49.927325 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.927111    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d5cd\" (UniqueName: \"kubernetes.io/projected/d16d3aa2-07ae-4ae8-a656-1471c7903458-kube-api-access-7d5cd\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:14:49.927325 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:49.927119    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d16d3aa2-07ae-4ae8-a656-1471c7903458-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:14:50.519997 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:50.519973    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"
Sep 17 17:14:50.520394 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:50.520001    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f" event={"ID":"d16d3aa2-07ae-4ae8-a656-1471c7903458","Type":"ContainerDied","Data":"b3a59f68d08812efdbd1942a9458a4604f645e089b546e29ba0a117c46cf6bd5"}
Sep 17 17:14:50.520394 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:50.520065    2577 scope.go:117] "RemoveContainer" containerID="54b211bc36ed47f2386115f1596b37cb4e248429993edcaaeb0e676952170cf1"
Sep 17 17:14:50.528197 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:50.528179    2577 scope.go:117] "RemoveContainer" containerID="4d7c0e6b8db0e349a81a11fdebb4023974e5504d599a565f9d8feb0a03272748"
Sep 17 17:14:50.535534 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:50.535517    2577 scope.go:117] "RemoveContainer" containerID="ad8106273412f410528cb30d4bd8e7ebcba12136f119fef4e9ff4660a0abd6a3"
Sep 17 17:14:50.540050 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:50.540029    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"]
Sep 17 17:14:50.542233 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:50.542210    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-d5q8f"]
Sep 17 17:14:52.446485 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:52.446449    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" path="/var/lib/kubelet/pods/d16d3aa2-07ae-4ae8-a656-1471c7903458/volumes"
Sep 17 17:14:56.386852 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:14:56.386803    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 17:15:06.386668 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:06.386627    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 17:15:16.387451 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:16.387412    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 17:15:26.387264 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:26.387212    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 17:15:30.446817 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:30.446791    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:15:33.750508 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:33.750466    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"]
Sep 17 17:15:33.750913 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:33.750720    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kserve-container" containerID="cri-o://ae2cf0e3670ab2d7a11ab4a486219915cb4316d5ee131335beb25c08d7888819" gracePeriod=30
Sep 17 17:15:33.750913 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:33.750775    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kube-rbac-proxy" containerID="cri-o://54b1b8b3f520253158bda0560137e72abeddebde0154aa8a2926c1e8247a7375" gracePeriod=30
Sep 17 17:15:34.716288 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:34.716254    2577 generic.go:356] "Generic (PLEG): container finished" podID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerID="54b1b8b3f520253158bda0560137e72abeddebde0154aa8a2926c1e8247a7375" exitCode=2
Sep 17 17:15:34.716450 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:34.716303    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" event={"ID":"9ddac1ea-192f-4b56-ab02-65f0fa3c6867","Type":"ContainerDied","Data":"54b1b8b3f520253158bda0560137e72abeddebde0154aa8a2926c1e8247a7375"}
Sep 17 17:15:36.382028 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:36.381988    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 17 17:15:38.004445 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.004421    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:15:38.091843 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.091784    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-proxy-tls\") pod \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") "
Sep 17 17:15:38.091960 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.091854    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kube-rbac-proxy-sar-config\") pod \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") "
Sep 17 17:15:38.091960 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.091906    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kserve-provision-location\") pod \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") "
Sep 17 17:15:38.091960 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.091932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4fxn\" (UniqueName: \"kubernetes.io/projected/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kube-api-access-l4fxn\") pod \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") "
Sep 17 17:15:38.092076 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.091966    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-cabundle-cert\") pod \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\" (UID: \"9ddac1ea-192f-4b56-ab02-65f0fa3c6867\") "
Sep 17 17:15:38.092255 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.092219    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9ddac1ea-192f-4b56-ab02-65f0fa3c6867" (UID: "9ddac1ea-192f-4b56-ab02-65f0fa3c6867"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:15:38.092381 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.092251    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9ddac1ea-192f-4b56-ab02-65f0fa3c6867" (UID: "9ddac1ea-192f-4b56-ab02-65f0fa3c6867"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:15:38.092430 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.092401    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9ddac1ea-192f-4b56-ab02-65f0fa3c6867" (UID: "9ddac1ea-192f-4b56-ab02-65f0fa3c6867"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:15:38.093906 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.093884    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kube-api-access-l4fxn" (OuterVolumeSpecName: "kube-api-access-l4fxn") pod "9ddac1ea-192f-4b56-ab02-65f0fa3c6867" (UID: "9ddac1ea-192f-4b56-ab02-65f0fa3c6867"). InnerVolumeSpecName "kube-api-access-l4fxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:15:38.093965 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.093913    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ddac1ea-192f-4b56-ab02-65f0fa3c6867" (UID: "9ddac1ea-192f-4b56-ab02-65f0fa3c6867"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:15:38.193258 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.193233    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:15:38.193258 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.193252    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4fxn\" (UniqueName: \"kubernetes.io/projected/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kube-api-access-l4fxn\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:15:38.193398 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.193263    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:15:38.193398 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.193290    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:15:38.193398 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.193299    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ddac1ea-192f-4b56-ab02-65f0fa3c6867-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:15:38.736060 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.735979    2577 generic.go:356] "Generic (PLEG): container finished" podID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerID="ae2cf0e3670ab2d7a11ab4a486219915cb4316d5ee131335beb25c08d7888819" exitCode=0
Sep 17 17:15:38.736060 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.736042    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"
Sep 17 17:15:38.736240 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.736063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" event={"ID":"9ddac1ea-192f-4b56-ab02-65f0fa3c6867","Type":"ContainerDied","Data":"ae2cf0e3670ab2d7a11ab4a486219915cb4316d5ee131335beb25c08d7888819"}
Sep 17 17:15:38.736240 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.736123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97" event={"ID":"9ddac1ea-192f-4b56-ab02-65f0fa3c6867","Type":"ContainerDied","Data":"a5e13752857006424d42f5a52359bc5ddf9c7494933422587a9105d62c973f42"}
Sep 17 17:15:38.736240 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.736151    2577 scope.go:117] "RemoveContainer" containerID="54b1b8b3f520253158bda0560137e72abeddebde0154aa8a2926c1e8247a7375"
Sep 17 17:15:38.744798 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.744780    2577 scope.go:117] "RemoveContainer" containerID="ae2cf0e3670ab2d7a11ab4a486219915cb4316d5ee131335beb25c08d7888819"
Sep 17 17:15:38.752308 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.752281    2577 scope.go:117] "RemoveContainer" containerID="794ba5e5b187883801f37083da2b55eadba62ac7a556c3f18d943250e612d59b"
Sep 17 17:15:38.753880 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.753860    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"]
Sep 17 17:15:38.760223 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.760198    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-wkr97"]
Sep 17 17:15:38.760490 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.760475    2577 scope.go:117] "RemoveContainer" containerID="54b1b8b3f520253158bda0560137e72abeddebde0154aa8a2926c1e8247a7375"
Sep 17 17:15:38.760765 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:15:38.760745    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b1b8b3f520253158bda0560137e72abeddebde0154aa8a2926c1e8247a7375\": container with ID starting with 54b1b8b3f520253158bda0560137e72abeddebde0154aa8a2926c1e8247a7375 not found: ID does not exist" containerID="54b1b8b3f520253158bda0560137e72abeddebde0154aa8a2926c1e8247a7375"
Sep 17 17:15:38.760842 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.760777    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b1b8b3f520253158bda0560137e72abeddebde0154aa8a2926c1e8247a7375"} err="failed to get container status \"54b1b8b3f520253158bda0560137e72abeddebde0154aa8a2926c1e8247a7375\": rpc error: code = NotFound desc = could not find container \"54b1b8b3f520253158bda0560137e72abeddebde0154aa8a2926c1e8247a7375\": container with ID starting with 54b1b8b3f520253158bda0560137e72abeddebde0154aa8a2926c1e8247a7375 not found: ID does not exist"
Sep 17 17:15:38.760842 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.760802    2577 scope.go:117] "RemoveContainer" containerID="ae2cf0e3670ab2d7a11ab4a486219915cb4316d5ee131335beb25c08d7888819"
Sep 17 17:15:38.761069 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:15:38.761053    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2cf0e3670ab2d7a11ab4a486219915cb4316d5ee131335beb25c08d7888819\": container with ID starting with ae2cf0e3670ab2d7a11ab4a486219915cb4316d5ee131335beb25c08d7888819 not found: ID does not exist" containerID="ae2cf0e3670ab2d7a11ab4a486219915cb4316d5ee131335beb25c08d7888819"
Sep 17 17:15:38.761108 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.761077    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2cf0e3670ab2d7a11ab4a486219915cb4316d5ee131335beb25c08d7888819"} err="failed to get container status \"ae2cf0e3670ab2d7a11ab4a486219915cb4316d5ee131335beb25c08d7888819\": rpc error: code = NotFound desc = could not find container \"ae2cf0e3670ab2d7a11ab4a486219915cb4316d5ee131335beb25c08d7888819\": container with ID starting with ae2cf0e3670ab2d7a11ab4a486219915cb4316d5ee131335beb25c08d7888819 not found: ID does not exist"
Sep 17 17:15:38.761108 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.761094    2577 scope.go:117] "RemoveContainer" containerID="794ba5e5b187883801f37083da2b55eadba62ac7a556c3f18d943250e612d59b"
Sep 17 17:15:38.761340 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:15:38.761322    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794ba5e5b187883801f37083da2b55eadba62ac7a556c3f18d943250e612d59b\": container with ID starting with 794ba5e5b187883801f37083da2b55eadba62ac7a556c3f18d943250e612d59b not found: ID does not exist" containerID="794ba5e5b187883801f37083da2b55eadba62ac7a556c3f18d943250e612d59b"
Sep 17 17:15:38.761401 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.761351    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794ba5e5b187883801f37083da2b55eadba62ac7a556c3f18d943250e612d59b"} err="failed to get container status \"794ba5e5b187883801f37083da2b55eadba62ac7a556c3f18d943250e612d59b\": rpc error: code = NotFound desc = could not find container \"794ba5e5b187883801f37083da2b55eadba62ac7a556c3f18d943250e612d59b\": container with ID starting with 794ba5e5b187883801f37083da2b55eadba62ac7a556c3f18d943250e612d59b not found: ID does not exist"
Sep 17 17:15:38.963083 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963050    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"]
Sep 17 17:15:38.963403 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963389    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="storage-initializer"
Sep 17 17:15:38.963460 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963404    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="storage-initializer"
Sep 17 17:15:38.963460 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963417    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kube-rbac-proxy"
Sep 17 17:15:38.963460 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963423    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kube-rbac-proxy"
Sep 17 17:15:38.963460 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963434    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kube-rbac-proxy"
Sep 17 17:15:38.963460 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963440    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kube-rbac-proxy"
Sep 17 17:15:38.963460 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963447    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="storage-initializer"
Sep 17 17:15:38.963460 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963453    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="storage-initializer"
Sep 17 17:15:38.963460 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963459    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kserve-container"
Sep 17 17:15:38.963695 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963464    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kserve-container"
Sep 17 17:15:38.963695 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963473    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kserve-container"
Sep 17 17:15:38.963695 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963477    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kserve-container"
Sep 17 17:15:38.963695 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963531    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kserve-container"
Sep 17 17:15:38.963695 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963540    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" containerName="kube-rbac-proxy"
Sep 17 17:15:38.963695 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963550    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kube-rbac-proxy"
Sep 17 17:15:38.963695 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.963555    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d16d3aa2-07ae-4ae8-a656-1471c7903458" containerName="kserve-container"
Sep 17 17:15:38.967885 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.967867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:38.970418 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.970390    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 17:15:38.970537 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.970467    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:15:38.970651 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.970632    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:15:38.970739 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.970698    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 17:15:38.970739 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.970714    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:15:38.970859 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.970761    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:15:38.970859 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.970715    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:15:38.977411 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:38.977387    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"]
Sep 17 17:15:39.000504 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.000448    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"]
Sep 17 17:15:39.003714 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.003699    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.006228 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.006212    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 17:15:39.013506 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.013485    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"]
Sep 17 17:15:39.099201 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.099151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q47w4\" (UniqueName: \"kubernetes.io/projected/3e498b3e-fea6-49f6-affe-c71688573d95-kube-api-access-q47w4\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-bpqwb\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.099297 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.099213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e498b3e-fea6-49f6-affe-c71688573d95-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-bpqwb\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.099297 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.099238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e498b3e-fea6-49f6-affe-c71688573d95-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-bpqwb\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.099297 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.099256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e498b3e-fea6-49f6-affe-c71688573d95-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-bpqwb\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.099401 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.099313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-cabundle-cert\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.099401 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.099347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck929\" (UniqueName: \"kubernetes.io/projected/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kube-api-access-ck929\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.099401 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.099373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.099500 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.099400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e498b3e-fea6-49f6-affe-c71688573d95-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-bpqwb\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.099500 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.099417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-proxy-tls\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.099500 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.099433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.200699 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.200676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e498b3e-fea6-49f6-affe-c71688573d95-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-bpqwb\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.200788 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.200704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-proxy-tls\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.200788 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.200723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.200788 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.200748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q47w4\" (UniqueName: \"kubernetes.io/projected/3e498b3e-fea6-49f6-affe-c71688573d95-kube-api-access-q47w4\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-bpqwb\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.200788 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.200769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e498b3e-fea6-49f6-affe-c71688573d95-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-bpqwb\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.201013 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.200790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e498b3e-fea6-49f6-affe-c71688573d95-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-bpqwb\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.201013 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.200805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e498b3e-fea6-49f6-affe-c71688573d95-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-bpqwb\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.201013 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:15:39.200812    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 17:15:39.201013 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.200849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-cabundle-cert\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.201013 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:15:39.200885    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-proxy-tls podName:d2f1db76-f1cf-460f-9ddc-36e7301f3d3f nodeName:}" failed. No retries permitted until 2026-09-17 17:15:39.700865848 +0000 UTC m=+2057.725774127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-proxy-tls") pod "isvc-canary-stop-predictor-67d6bcd4b5-m58rv" (UID: "d2f1db76-f1cf-460f-9ddc-36e7301f3d3f") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 17:15:39.201013 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.200920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck929\" (UniqueName: \"kubernetes.io/projected/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kube-api-access-ck929\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.201013 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.200952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.201481 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.201343    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.201548 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.201518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e498b3e-fea6-49f6-affe-c71688573d95-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-bpqwb\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.201664 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.201631    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.201664 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.201660    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-cabundle-cert\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.201815 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.201673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e498b3e-fea6-49f6-affe-c71688573d95-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-bpqwb\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.201987 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.201965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e498b3e-fea6-49f6-affe-c71688573d95-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-bpqwb\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.203379 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.203360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e498b3e-fea6-49f6-affe-c71688573d95-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-bpqwb\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.209899 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.209871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck929\" (UniqueName: \"kubernetes.io/projected/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kube-api-access-ck929\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.209997 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.209983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q47w4\" (UniqueName: \"kubernetes.io/projected/3e498b3e-fea6-49f6-affe-c71688573d95-kube-api-access-q47w4\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-bpqwb\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.278409 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.278353    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:39.554721 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.554697    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"]
Sep 17 17:15:39.556985 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:15:39.556956    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e498b3e_fea6_49f6_affe_c71688573d95.slice/crio-5bb4c85b1c299d19be829f9d91b6228f5e796581f0a70396c3309e0614878132 WatchSource:0}: Error finding container 5bb4c85b1c299d19be829f9d91b6228f5e796581f0a70396c3309e0614878132: Status 404 returned error can't find the container with id 5bb4c85b1c299d19be829f9d91b6228f5e796581f0a70396c3309e0614878132
Sep 17 17:15:39.705497 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.705465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-proxy-tls\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.707589 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.707569    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-proxy-tls\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-m58rv\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:39.742713 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.742683    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" event={"ID":"3e498b3e-fea6-49f6-affe-c71688573d95","Type":"ContainerStarted","Data":"cef81bbe3c243bf7f4ce61f2c244d477e05f6af86ccff29bbe00f30b1b19cccd"}
Sep 17 17:15:39.742842 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.742740    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" event={"ID":"3e498b3e-fea6-49f6-affe-c71688573d95","Type":"ContainerStarted","Data":"5bb4c85b1c299d19be829f9d91b6228f5e796581f0a70396c3309e0614878132"}
Sep 17 17:15:39.914080 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:39.914008    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:40.186506 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:40.186356    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"]
Sep 17 17:15:40.188427 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:15:40.188402    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f1db76_f1cf_460f_9ddc_36e7301f3d3f.slice/crio-1bb6c1d9b25267be24c74ce8b39e004c8c61801a245f9ad25b4ad387a90b50a6 WatchSource:0}: Error finding container 1bb6c1d9b25267be24c74ce8b39e004c8c61801a245f9ad25b4ad387a90b50a6: Status 404 returned error can't find the container with id 1bb6c1d9b25267be24c74ce8b39e004c8c61801a245f9ad25b4ad387a90b50a6
Sep 17 17:15:40.446509 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:40.446431    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ddac1ea-192f-4b56-ab02-65f0fa3c6867" path="/var/lib/kubelet/pods/9ddac1ea-192f-4b56-ab02-65f0fa3c6867/volumes"
Sep 17 17:15:40.748481 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:40.748397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" event={"ID":"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f","Type":"ContainerStarted","Data":"d14384c930a3c015e2a715a9dc5579a2161aaf33a213a3b35f83525b4f2a320a"}
Sep 17 17:15:40.748481 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:40.748459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" event={"ID":"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f","Type":"ContainerStarted","Data":"1bb6c1d9b25267be24c74ce8b39e004c8c61801a245f9ad25b4ad387a90b50a6"}
Sep 17 17:15:43.768980 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:43.768896    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e498b3e-fea6-49f6-affe-c71688573d95" containerID="cef81bbe3c243bf7f4ce61f2c244d477e05f6af86ccff29bbe00f30b1b19cccd" exitCode=0
Sep 17 17:15:43.768980 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:43.768968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" event={"ID":"3e498b3e-fea6-49f6-affe-c71688573d95","Type":"ContainerDied","Data":"cef81bbe3c243bf7f4ce61f2c244d477e05f6af86ccff29bbe00f30b1b19cccd"}
Sep 17 17:15:44.775357 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:44.775319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" event={"ID":"3e498b3e-fea6-49f6-affe-c71688573d95","Type":"ContainerStarted","Data":"7f25f9fb693417fa37707415761c040b3be39a1e92bf330e8bc414ab7a5298da"}
Sep 17 17:15:44.775753 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:44.775384    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" event={"ID":"3e498b3e-fea6-49f6-affe-c71688573d95","Type":"ContainerStarted","Data":"5a575816f6fa3f8a417fc893f4d9e66a2c0df7717b17e19b5f1f5a07c7c5723b"}
Sep 17 17:15:44.775753 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:44.775568    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:44.776820 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:44.776801    2577 generic.go:356] "Generic (PLEG): container finished" podID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerID="d14384c930a3c015e2a715a9dc5579a2161aaf33a213a3b35f83525b4f2a320a" exitCode=0
Sep 17 17:15:44.776879 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:44.776859    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" event={"ID":"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f","Type":"ContainerDied","Data":"d14384c930a3c015e2a715a9dc5579a2161aaf33a213a3b35f83525b4f2a320a"}
Sep 17 17:15:44.797086 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:44.797044    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" podStartSLOduration=3.104453091 podStartE2EDuration="6.797031714s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.692578623s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:15:44.794952041 +0000 UTC m=+2062.819860330" watchObservedRunningTime="2026-09-17 17:15:44.797031714 +0000 UTC m=+2062.821940001"
Sep 17 17:15:45.783881 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:45.783841    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" event={"ID":"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f","Type":"ContainerStarted","Data":"145e181bc3ba7bbf7c832b57599b1e5d2ee8b62821944a9a40e431f62248e977"}
Sep 17 17:15:45.784273 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:45.783912    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" event={"ID":"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f","Type":"ContainerStarted","Data":"108936d775ade320735a4de8c5dfd391441818e557e2ceb6eb6678cf2cf24e34"}
Sep 17 17:15:45.784273 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:45.784142    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:45.784462 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:45.784438    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:45.784624 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:45.784600    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:45.785527 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:45.785502    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 17:15:45.785674 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:45.785646    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 17:15:46.788179 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:46.788111    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 17:15:46.788625 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:46.788120    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 17:15:47.793006 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:47.792968    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 17:15:51.792943 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:51.792914    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:15:51.793528 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:51.793497    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 17:15:51.813522 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:51.813465    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" podStartSLOduration=10.108158574 podStartE2EDuration="13.813449546s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.705290972s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:15:45.804695694 +0000 UTC m=+2063.829604012" watchObservedRunningTime="2026-09-17 17:15:51.813449546 +0000 UTC m=+2069.838357837"
Sep 17 17:15:52.797477 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:52.797443    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:15:52.797956 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:15:52.797930    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 17:16:01.793811 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:01.793771    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 17:16:02.798075 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:02.798032    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 17:16:11.794398 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:11.794355    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 17:16:12.798911 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:12.798867    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 17:16:21.793985 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:21.793947    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 17:16:22.559584 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:22.559552    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 17:16:22.563277 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:22.563252    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 17:16:22.663144 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:22.663117    2577 scope.go:117] "RemoveContainer" containerID="9a64af9c604dcb456e227786fc1048450f24b03ec4a7cb05f2f45044ff6b7254"
Sep 17 17:16:22.670564 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:22.670547    2577 scope.go:117] "RemoveContainer" containerID="8a3098a51fa2502d1697fbd25b730934491c92e3736677b89d46dd0f9087673e"
Sep 17 17:16:22.677372 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:22.677359    2577 scope.go:117] "RemoveContainer" containerID="85cd85eb436cc1be59923714b47e6aa6a79ea59868bbbf153a87c6d7669ea436"
Sep 17 17:16:22.797928 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:22.797895    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 17:16:31.793941 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:31.793902    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 17:16:32.798803 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:32.798761    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 17:16:41.794346 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:41.794304    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 17:16:42.797898 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:42.797856    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 17:16:51.793403 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:51.793361    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 17:16:52.798819 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:16:52.798777    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 17:17:01.794887 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:01.794841    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:17:02.798318 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:02.798288    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:17:08.956512 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:08.956475    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"]
Sep 17 17:17:08.956914 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:08.956772    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kserve-container" containerID="cri-o://5a575816f6fa3f8a417fc893f4d9e66a2c0df7717b17e19b5f1f5a07c7c5723b" gracePeriod=30
Sep 17 17:17:08.956914 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:08.956810    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kube-rbac-proxy" containerID="cri-o://7f25f9fb693417fa37707415761c040b3be39a1e92bf330e8bc414ab7a5298da" gracePeriod=30
Sep 17 17:17:09.021433 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:09.021405    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"]
Sep 17 17:17:09.021690 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:09.021666    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kserve-container" containerID="cri-o://108936d775ade320735a4de8c5dfd391441818e557e2ceb6eb6678cf2cf24e34" gracePeriod=30
Sep 17 17:17:09.021796 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:09.021683    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kube-rbac-proxy" containerID="cri-o://145e181bc3ba7bbf7c832b57599b1e5d2ee8b62821944a9a40e431f62248e977" gracePeriod=30
Sep 17 17:17:09.180959 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:09.180931    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e498b3e-fea6-49f6-affe-c71688573d95" containerID="7f25f9fb693417fa37707415761c040b3be39a1e92bf330e8bc414ab7a5298da" exitCode=2
Sep 17 17:17:09.181061 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:09.181004    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" event={"ID":"3e498b3e-fea6-49f6-affe-c71688573d95","Type":"ContainerDied","Data":"7f25f9fb693417fa37707415761c040b3be39a1e92bf330e8bc414ab7a5298da"}
Sep 17 17:17:09.183269 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:09.183250    2577 generic.go:356] "Generic (PLEG): container finished" podID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerID="145e181bc3ba7bbf7c832b57599b1e5d2ee8b62821944a9a40e431f62248e977" exitCode=2
Sep 17 17:17:09.183355 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:09.183308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" event={"ID":"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f","Type":"ContainerDied","Data":"145e181bc3ba7bbf7c832b57599b1e5d2ee8b62821944a9a40e431f62248e977"}
Sep 17 17:17:11.788649 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:11.788603    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 17 17:17:11.793514 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:11.793491    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 17:17:12.793486 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:12.793440    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 17 17:17:12.798627 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:12.798587    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 17:17:13.206370 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.206338    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e498b3e-fea6-49f6-affe-c71688573d95" containerID="5a575816f6fa3f8a417fc893f4d9e66a2c0df7717b17e19b5f1f5a07c7c5723b" exitCode=0
Sep 17 17:17:13.206533 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.206406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" event={"ID":"3e498b3e-fea6-49f6-affe-c71688573d95","Type":"ContainerDied","Data":"5a575816f6fa3f8a417fc893f4d9e66a2c0df7717b17e19b5f1f5a07c7c5723b"}
Sep 17 17:17:13.209341 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.209317    2577 generic.go:356] "Generic (PLEG): container finished" podID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerID="108936d775ade320735a4de8c5dfd391441818e557e2ceb6eb6678cf2cf24e34" exitCode=0
Sep 17 17:17:13.209478 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.209374    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" event={"ID":"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f","Type":"ContainerDied","Data":"108936d775ade320735a4de8c5dfd391441818e557e2ceb6eb6678cf2cf24e34"}
Sep 17 17:17:13.320281 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.320260    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:17:13.323080 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.323061    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:17:13.408678 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.408623    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck929\" (UniqueName: \"kubernetes.io/projected/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kube-api-access-ck929\") pod \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") "
Sep 17 17:17:13.408678 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.408668    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e498b3e-fea6-49f6-affe-c71688573d95-proxy-tls\") pod \"3e498b3e-fea6-49f6-affe-c71688573d95\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") "
Sep 17 17:17:13.408846 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.408698    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q47w4\" (UniqueName: \"kubernetes.io/projected/3e498b3e-fea6-49f6-affe-c71688573d95-kube-api-access-q47w4\") pod \"3e498b3e-fea6-49f6-affe-c71688573d95\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") "
Sep 17 17:17:13.408846 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.408715    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kserve-provision-location\") pod \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") "
Sep 17 17:17:13.408846 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.408737    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e498b3e-fea6-49f6-affe-c71688573d95-kserve-provision-location\") pod \"3e498b3e-fea6-49f6-affe-c71688573d95\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") "
Sep 17 17:17:13.408846 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.408755    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-proxy-tls\") pod \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") "
Sep 17 17:17:13.408846 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.408775    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-cabundle-cert\") pod \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") "
Sep 17 17:17:13.408846 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.408790    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kube-rbac-proxy-sar-config\") pod \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\" (UID: \"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f\") "
Sep 17 17:17:13.408846 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.408819    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e498b3e-fea6-49f6-affe-c71688573d95-cabundle-cert\") pod \"3e498b3e-fea6-49f6-affe-c71688573d95\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") "
Sep 17 17:17:13.408846 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.408846    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e498b3e-fea6-49f6-affe-c71688573d95-kube-rbac-proxy-sar-config\") pod \"3e498b3e-fea6-49f6-affe-c71688573d95\" (UID: \"3e498b3e-fea6-49f6-affe-c71688573d95\") "
Sep 17 17:17:13.409298 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.409072    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e498b3e-fea6-49f6-affe-c71688573d95-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e498b3e-fea6-49f6-affe-c71688573d95" (UID: "3e498b3e-fea6-49f6-affe-c71688573d95"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:17:13.409360 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.409249    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" (UID: "d2f1db76-f1cf-460f-9ddc-36e7301f3d3f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:17:13.409423 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.409363    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" (UID: "d2f1db76-f1cf-460f-9ddc-36e7301f3d3f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:17:13.409483 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.409419    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" (UID: "d2f1db76-f1cf-460f-9ddc-36e7301f3d3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:17:13.409539 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.409491    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e498b3e-fea6-49f6-affe-c71688573d95-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e498b3e-fea6-49f6-affe-c71688573d95" (UID: "3e498b3e-fea6-49f6-affe-c71688573d95"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:17:13.409679 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.409657    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e498b3e-fea6-49f6-affe-c71688573d95-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e498b3e-fea6-49f6-affe-c71688573d95" (UID: "3e498b3e-fea6-49f6-affe-c71688573d95"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:17:13.410910 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.410882    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e498b3e-fea6-49f6-affe-c71688573d95-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e498b3e-fea6-49f6-affe-c71688573d95" (UID: "3e498b3e-fea6-49f6-affe-c71688573d95"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:17:13.410910 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.410887    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" (UID: "d2f1db76-f1cf-460f-9ddc-36e7301f3d3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:17:13.411209 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.411192    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e498b3e-fea6-49f6-affe-c71688573d95-kube-api-access-q47w4" (OuterVolumeSpecName: "kube-api-access-q47w4") pod "3e498b3e-fea6-49f6-affe-c71688573d95" (UID: "3e498b3e-fea6-49f6-affe-c71688573d95"). InnerVolumeSpecName "kube-api-access-q47w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:17:13.411300 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.411286    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kube-api-access-ck929" (OuterVolumeSpecName: "kube-api-access-ck929") pod "d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" (UID: "d2f1db76-f1cf-460f-9ddc-36e7301f3d3f"). InnerVolumeSpecName "kube-api-access-ck929". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:17:13.509890 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.509867    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e498b3e-fea6-49f6-affe-c71688573d95-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:17:13.509890 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.509887    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q47w4\" (UniqueName: \"kubernetes.io/projected/3e498b3e-fea6-49f6-affe-c71688573d95-kube-api-access-q47w4\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:17:13.510012 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.509896    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:17:13.510012 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.509906    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e498b3e-fea6-49f6-affe-c71688573d95-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:17:13.510012 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.509916    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:17:13.510012 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.509924    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:17:13.510012 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.509931    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:17:13.510012 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.509939    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e498b3e-fea6-49f6-affe-c71688573d95-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:17:13.510012 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.509947    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e498b3e-fea6-49f6-affe-c71688573d95-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:17:13.510012 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:13.509955    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck929\" (UniqueName: \"kubernetes.io/projected/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f-kube-api-access-ck929\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:17:14.217989 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.217952    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"
Sep 17 17:17:14.218521 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.217952    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv" event={"ID":"d2f1db76-f1cf-460f-9ddc-36e7301f3d3f","Type":"ContainerDied","Data":"1bb6c1d9b25267be24c74ce8b39e004c8c61801a245f9ad25b4ad387a90b50a6"}
Sep 17 17:17:14.218521 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.218215    2577 scope.go:117] "RemoveContainer" containerID="145e181bc3ba7bbf7c832b57599b1e5d2ee8b62821944a9a40e431f62248e977"
Sep 17 17:17:14.220635 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.220601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb" event={"ID":"3e498b3e-fea6-49f6-affe-c71688573d95","Type":"ContainerDied","Data":"5bb4c85b1c299d19be829f9d91b6228f5e796581f0a70396c3309e0614878132"}
Sep 17 17:17:14.220750 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.220654    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"
Sep 17 17:17:14.229041 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.229013    2577 scope.go:117] "RemoveContainer" containerID="108936d775ade320735a4de8c5dfd391441818e557e2ceb6eb6678cf2cf24e34"
Sep 17 17:17:14.237141 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.237125    2577 scope.go:117] "RemoveContainer" containerID="d14384c930a3c015e2a715a9dc5579a2161aaf33a213a3b35f83525b4f2a320a"
Sep 17 17:17:14.242284 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.242260    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"]
Sep 17 17:17:14.245300 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.245269    2577 scope.go:117] "RemoveContainer" containerID="7f25f9fb693417fa37707415761c040b3be39a1e92bf330e8bc414ab7a5298da"
Sep 17 17:17:14.247071 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.247045    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-m58rv"]
Sep 17 17:17:14.253085 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.253065    2577 scope.go:117] "RemoveContainer" containerID="5a575816f6fa3f8a417fc893f4d9e66a2c0df7717b17e19b5f1f5a07c7c5723b"
Sep 17 17:17:14.257940 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.257917    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"]
Sep 17 17:17:14.260916 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.260895    2577 scope.go:117] "RemoveContainer" containerID="cef81bbe3c243bf7f4ce61f2c244d477e05f6af86ccff29bbe00f30b1b19cccd"
Sep 17 17:17:14.262307 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.262285    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-bpqwb"]
Sep 17 17:17:14.447540 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.447474    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" path="/var/lib/kubelet/pods/3e498b3e-fea6-49f6-affe-c71688573d95/volumes"
Sep 17 17:17:14.448010 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:14.447990    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" path="/var/lib/kubelet/pods/d2f1db76-f1cf-460f-9ddc-36e7301f3d3f/volumes"
Sep 17 17:17:19.222659 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.222585    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"]
Sep 17 17:17:19.222998 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.222907    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kserve-container"
Sep 17 17:17:19.222998 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.222918    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kserve-container"
Sep 17 17:17:19.222998 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.222931    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="storage-initializer"
Sep 17 17:17:19.222998 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.222936    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="storage-initializer"
Sep 17 17:17:19.222998 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.222950    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kube-rbac-proxy"
Sep 17 17:17:19.222998 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.222955    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kube-rbac-proxy"
Sep 17 17:17:19.222998 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.222961    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="storage-initializer"
Sep 17 17:17:19.222998 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.222967    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="storage-initializer"
Sep 17 17:17:19.222998 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.222975    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kserve-container"
Sep 17 17:17:19.222998 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.222981    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kserve-container"
Sep 17 17:17:19.222998 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.222989    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kube-rbac-proxy"
Sep 17 17:17:19.222998 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.222994    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kube-rbac-proxy"
Sep 17 17:17:19.223439 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.223042    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kube-rbac-proxy"
Sep 17 17:17:19.223439 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.223050    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e498b3e-fea6-49f6-affe-c71688573d95" containerName="kserve-container"
Sep 17 17:17:19.223439 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.223056    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kserve-container"
Sep 17 17:17:19.223439 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.223062    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f1db76-f1cf-460f-9ddc-36e7301f3d3f" containerName="kube-rbac-proxy"
Sep 17 17:17:19.227886 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.227866    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.230611 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.230591    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:17:19.230611 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.230604    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:17:19.230924 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.230909    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:17:19.231771 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.231755    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 17:17:19.231857 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.231819    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 17 17:17:19.231902 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.231886    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:17:19.231957 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.231938    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:17:19.235298 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.235025    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"]
Sep 17 17:17:19.251985 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.251961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a825dd20-972f-4f5e-80aa-77db164b9036-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-mmkv8\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.252069 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.252001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a825dd20-972f-4f5e-80aa-77db164b9036-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-mmkv8\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.252069 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.252022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a825dd20-972f-4f5e-80aa-77db164b9036-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-mmkv8\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.252210 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.252122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx46f\" (UniqueName: \"kubernetes.io/projected/a825dd20-972f-4f5e-80aa-77db164b9036-kube-api-access-lx46f\") pod \"isvc-lightgbm-predictor-54f85ddf88-mmkv8\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.252210 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.252172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a825dd20-972f-4f5e-80aa-77db164b9036-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-mmkv8\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.353352 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.353326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx46f\" (UniqueName: \"kubernetes.io/projected/a825dd20-972f-4f5e-80aa-77db164b9036-kube-api-access-lx46f\") pod \"isvc-lightgbm-predictor-54f85ddf88-mmkv8\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.353453 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.353359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a825dd20-972f-4f5e-80aa-77db164b9036-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-mmkv8\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.353453 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.353432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a825dd20-972f-4f5e-80aa-77db164b9036-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-mmkv8\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.353566 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.353473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a825dd20-972f-4f5e-80aa-77db164b9036-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-mmkv8\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.353566 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.353503    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a825dd20-972f-4f5e-80aa-77db164b9036-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-mmkv8\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.353891 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.353871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a825dd20-972f-4f5e-80aa-77db164b9036-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-mmkv8\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.354077 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.354059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a825dd20-972f-4f5e-80aa-77db164b9036-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-mmkv8\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.354151 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.354133    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a825dd20-972f-4f5e-80aa-77db164b9036-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-mmkv8\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.356008 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.355991    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a825dd20-972f-4f5e-80aa-77db164b9036-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-mmkv8\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.360886 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.360861    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx46f\" (UniqueName: \"kubernetes.io/projected/a825dd20-972f-4f5e-80aa-77db164b9036-kube-api-access-lx46f\") pod \"isvc-lightgbm-predictor-54f85ddf88-mmkv8\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.538980 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.538927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:19.813021 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:19.812996    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"]
Sep 17 17:17:19.814686 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:17:19.814660    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda825dd20_972f_4f5e_80aa_77db164b9036.slice/crio-0f96369a9687a1be64a3788f829e415e7ebb7508d141cb5458d63dc76aa8f7e9 WatchSource:0}: Error finding container 0f96369a9687a1be64a3788f829e415e7ebb7508d141cb5458d63dc76aa8f7e9: Status 404 returned error can't find the container with id 0f96369a9687a1be64a3788f829e415e7ebb7508d141cb5458d63dc76aa8f7e9
Sep 17 17:17:20.255324 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:20.255239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" event={"ID":"a825dd20-972f-4f5e-80aa-77db164b9036","Type":"ContainerStarted","Data":"6f3e571c0997007a2645174777a2c6d5bfa2d1bb9fd25efee7cdbabb972bcf69"}
Sep 17 17:17:20.255324 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:20.255295    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" event={"ID":"a825dd20-972f-4f5e-80aa-77db164b9036","Type":"ContainerStarted","Data":"0f96369a9687a1be64a3788f829e415e7ebb7508d141cb5458d63dc76aa8f7e9"}
Sep 17 17:17:24.277107 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:24.277064    2577 generic.go:356] "Generic (PLEG): container finished" podID="a825dd20-972f-4f5e-80aa-77db164b9036" containerID="6f3e571c0997007a2645174777a2c6d5bfa2d1bb9fd25efee7cdbabb972bcf69" exitCode=0
Sep 17 17:17:24.277555 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:24.277138    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" event={"ID":"a825dd20-972f-4f5e-80aa-77db164b9036","Type":"ContainerDied","Data":"6f3e571c0997007a2645174777a2c6d5bfa2d1bb9fd25efee7cdbabb972bcf69"}
Sep 17 17:17:31.316338 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:31.316297    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" event={"ID":"a825dd20-972f-4f5e-80aa-77db164b9036","Type":"ContainerStarted","Data":"c2366756808493a3800072fd24c696aa58a414b89f47ffb655ce66d47a8fad84"}
Sep 17 17:17:31.316780 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:31.316358    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" event={"ID":"a825dd20-972f-4f5e-80aa-77db164b9036","Type":"ContainerStarted","Data":"88b2d55a8b0182d9c5fe782602bb7e37799ffb601d04d23f3953c4871028d598"}
Sep 17 17:17:31.316780 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:31.316547    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:31.335987 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:31.335942    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" podStartSLOduration=2.090810989 podStartE2EDuration="12.33592756s" totalImagesPullingTime="6.362221769s" totalInitContainerRuntime="3.882894802s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:17:31.333610578 +0000 UTC m=+2169.358518867" watchObservedRunningTime="2026-09-17 17:17:31.33592756 +0000 UTC m=+2169.360835849"
Sep 17 17:17:32.320772 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:32.320738    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:32.322000 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:32.321967    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 17:17:33.325009 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:33.324963    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 17:17:38.330533 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:38.330504    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:17:38.331051 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:38.330961    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 17:17:48.331381 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:48.331336    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 17:17:58.331592 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:17:58.331547    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 17:18:08.331462 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:18:08.331422    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 17:18:18.331479 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:18:18.331438    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 17:18:28.331633 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:18:28.331590    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 17:18:38.331696 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:18:38.331655    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 17:18:46.441976 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:18:46.441882    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 17:18:56.446710 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:18:56.446683    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:18:59.910631 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:18:59.910598    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"]
Sep 17 17:18:59.911050 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:18:59.910830    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kserve-container" containerID="cri-o://88b2d55a8b0182d9c5fe782602bb7e37799ffb601d04d23f3953c4871028d598" gracePeriod=30
Sep 17 17:18:59.911050 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:18:59.910885    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kube-rbac-proxy" containerID="cri-o://c2366756808493a3800072fd24c696aa58a414b89f47ffb655ce66d47a8fad84" gracePeriod=30
Sep 17 17:19:00.701730 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:00.701697    2577 generic.go:356] "Generic (PLEG): container finished" podID="a825dd20-972f-4f5e-80aa-77db164b9036" containerID="c2366756808493a3800072fd24c696aa58a414b89f47ffb655ce66d47a8fad84" exitCode=2
Sep 17 17:19:00.701893 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:00.701753    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" event={"ID":"a825dd20-972f-4f5e-80aa-77db164b9036","Type":"ContainerDied","Data":"c2366756808493a3800072fd24c696aa58a414b89f47ffb655ce66d47a8fad84"}
Sep 17 17:19:03.325898 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:03.325854    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 17 17:19:04.654397 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.654371    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:19:04.723438 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.723405    2577 generic.go:356] "Generic (PLEG): container finished" podID="a825dd20-972f-4f5e-80aa-77db164b9036" containerID="88b2d55a8b0182d9c5fe782602bb7e37799ffb601d04d23f3953c4871028d598" exitCode=0
Sep 17 17:19:04.723568 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.723448    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"
Sep 17 17:19:04.723568 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.723467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" event={"ID":"a825dd20-972f-4f5e-80aa-77db164b9036","Type":"ContainerDied","Data":"88b2d55a8b0182d9c5fe782602bb7e37799ffb601d04d23f3953c4871028d598"}
Sep 17 17:19:04.723568 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.723538    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8" event={"ID":"a825dd20-972f-4f5e-80aa-77db164b9036","Type":"ContainerDied","Data":"0f96369a9687a1be64a3788f829e415e7ebb7508d141cb5458d63dc76aa8f7e9"}
Sep 17 17:19:04.723695 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.723577    2577 scope.go:117] "RemoveContainer" containerID="c2366756808493a3800072fd24c696aa58a414b89f47ffb655ce66d47a8fad84"
Sep 17 17:19:04.725653 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.725636    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a825dd20-972f-4f5e-80aa-77db164b9036-kserve-provision-location\") pod \"a825dd20-972f-4f5e-80aa-77db164b9036\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") "
Sep 17 17:19:04.725730 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.725699    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a825dd20-972f-4f5e-80aa-77db164b9036-kube-rbac-proxy-sar-config\") pod \"a825dd20-972f-4f5e-80aa-77db164b9036\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") "
Sep 17 17:19:04.725775 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.725758    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx46f\" (UniqueName: \"kubernetes.io/projected/a825dd20-972f-4f5e-80aa-77db164b9036-kube-api-access-lx46f\") pod \"a825dd20-972f-4f5e-80aa-77db164b9036\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") "
Sep 17 17:19:04.725828 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.725786    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a825dd20-972f-4f5e-80aa-77db164b9036-cabundle-cert\") pod \"a825dd20-972f-4f5e-80aa-77db164b9036\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") "
Sep 17 17:19:04.725828 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.725814    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a825dd20-972f-4f5e-80aa-77db164b9036-proxy-tls\") pod \"a825dd20-972f-4f5e-80aa-77db164b9036\" (UID: \"a825dd20-972f-4f5e-80aa-77db164b9036\") "
Sep 17 17:19:04.726026 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.726001    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a825dd20-972f-4f5e-80aa-77db164b9036-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a825dd20-972f-4f5e-80aa-77db164b9036" (UID: "a825dd20-972f-4f5e-80aa-77db164b9036"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:19:04.726134 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.726112    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a825dd20-972f-4f5e-80aa-77db164b9036-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a825dd20-972f-4f5e-80aa-77db164b9036" (UID: "a825dd20-972f-4f5e-80aa-77db164b9036"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:19:04.726238 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.726207    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a825dd20-972f-4f5e-80aa-77db164b9036-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a825dd20-972f-4f5e-80aa-77db164b9036" (UID: "a825dd20-972f-4f5e-80aa-77db164b9036"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:19:04.727849 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.727829    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a825dd20-972f-4f5e-80aa-77db164b9036-kube-api-access-lx46f" (OuterVolumeSpecName: "kube-api-access-lx46f") pod "a825dd20-972f-4f5e-80aa-77db164b9036" (UID: "a825dd20-972f-4f5e-80aa-77db164b9036"). InnerVolumeSpecName "kube-api-access-lx46f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:19:04.728220 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.728202    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a825dd20-972f-4f5e-80aa-77db164b9036-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a825dd20-972f-4f5e-80aa-77db164b9036" (UID: "a825dd20-972f-4f5e-80aa-77db164b9036"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:19:04.731608 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.731582    2577 scope.go:117] "RemoveContainer" containerID="88b2d55a8b0182d9c5fe782602bb7e37799ffb601d04d23f3953c4871028d598"
Sep 17 17:19:04.741385 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.741367    2577 scope.go:117] "RemoveContainer" containerID="6f3e571c0997007a2645174777a2c6d5bfa2d1bb9fd25efee7cdbabb972bcf69"
Sep 17 17:19:04.748005 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.747989    2577 scope.go:117] "RemoveContainer" containerID="c2366756808493a3800072fd24c696aa58a414b89f47ffb655ce66d47a8fad84"
Sep 17 17:19:04.748264 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:19:04.748243    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2366756808493a3800072fd24c696aa58a414b89f47ffb655ce66d47a8fad84\": container with ID starting with c2366756808493a3800072fd24c696aa58a414b89f47ffb655ce66d47a8fad84 not found: ID does not exist" containerID="c2366756808493a3800072fd24c696aa58a414b89f47ffb655ce66d47a8fad84"
Sep 17 17:19:04.748345 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.748275    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2366756808493a3800072fd24c696aa58a414b89f47ffb655ce66d47a8fad84"} err="failed to get container status \"c2366756808493a3800072fd24c696aa58a414b89f47ffb655ce66d47a8fad84\": rpc error: code = NotFound desc = could not find container \"c2366756808493a3800072fd24c696aa58a414b89f47ffb655ce66d47a8fad84\": container with ID starting with c2366756808493a3800072fd24c696aa58a414b89f47ffb655ce66d47a8fad84 not found: ID does not exist"
Sep 17 17:19:04.748345 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.748298    2577 scope.go:117] "RemoveContainer" containerID="88b2d55a8b0182d9c5fe782602bb7e37799ffb601d04d23f3953c4871028d598"
Sep 17 17:19:04.748558 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:19:04.748541    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b2d55a8b0182d9c5fe782602bb7e37799ffb601d04d23f3953c4871028d598\": container with ID starting with 88b2d55a8b0182d9c5fe782602bb7e37799ffb601d04d23f3953c4871028d598 not found: ID does not exist" containerID="88b2d55a8b0182d9c5fe782602bb7e37799ffb601d04d23f3953c4871028d598"
Sep 17 17:19:04.748598 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.748565    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b2d55a8b0182d9c5fe782602bb7e37799ffb601d04d23f3953c4871028d598"} err="failed to get container status \"88b2d55a8b0182d9c5fe782602bb7e37799ffb601d04d23f3953c4871028d598\": rpc error: code = NotFound desc = could not find container \"88b2d55a8b0182d9c5fe782602bb7e37799ffb601d04d23f3953c4871028d598\": container with ID starting with 88b2d55a8b0182d9c5fe782602bb7e37799ffb601d04d23f3953c4871028d598 not found: ID does not exist"
Sep 17 17:19:04.748598 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.748580    2577 scope.go:117] "RemoveContainer" containerID="6f3e571c0997007a2645174777a2c6d5bfa2d1bb9fd25efee7cdbabb972bcf69"
Sep 17 17:19:04.748801 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:19:04.748783    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f3e571c0997007a2645174777a2c6d5bfa2d1bb9fd25efee7cdbabb972bcf69\": container with ID starting with 6f3e571c0997007a2645174777a2c6d5bfa2d1bb9fd25efee7cdbabb972bcf69 not found: ID does not exist" containerID="6f3e571c0997007a2645174777a2c6d5bfa2d1bb9fd25efee7cdbabb972bcf69"
Sep 17 17:19:04.748845 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.748806    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f3e571c0997007a2645174777a2c6d5bfa2d1bb9fd25efee7cdbabb972bcf69"} err="failed to get container status \"6f3e571c0997007a2645174777a2c6d5bfa2d1bb9fd25efee7cdbabb972bcf69\": rpc error: code = NotFound desc = could not find container \"6f3e571c0997007a2645174777a2c6d5bfa2d1bb9fd25efee7cdbabb972bcf69\": container with ID starting with 6f3e571c0997007a2645174777a2c6d5bfa2d1bb9fd25efee7cdbabb972bcf69 not found: ID does not exist"
Sep 17 17:19:04.827112 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.827063    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a825dd20-972f-4f5e-80aa-77db164b9036-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:19:04.827112 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.827084    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a825dd20-972f-4f5e-80aa-77db164b9036-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:19:04.827112 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.827097    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx46f\" (UniqueName: \"kubernetes.io/projected/a825dd20-972f-4f5e-80aa-77db164b9036-kube-api-access-lx46f\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:19:04.827112 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.827111    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a825dd20-972f-4f5e-80aa-77db164b9036-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:19:04.827325 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:04.827123    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a825dd20-972f-4f5e-80aa-77db164b9036-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:19:05.048238 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.048214    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"]
Sep 17 17:19:05.053911 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.053887    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-mmkv8"]
Sep 17 17:19:05.137030 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.136967    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"]
Sep 17 17:19:05.137347 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.137330    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kserve-container"
Sep 17 17:19:05.137421 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.137349    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kserve-container"
Sep 17 17:19:05.137421 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.137368    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="storage-initializer"
Sep 17 17:19:05.137421 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.137377    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="storage-initializer"
Sep 17 17:19:05.137421 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.137407    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kube-rbac-proxy"
Sep 17 17:19:05.137421 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.137415    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kube-rbac-proxy"
Sep 17 17:19:05.137688 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.137498    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kserve-container"
Sep 17 17:19:05.137688 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.137515    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" containerName="kube-rbac-proxy"
Sep 17 17:19:05.142224 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.142206    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.144745 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.144721    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:19:05.144852 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.144745    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 17 17:19:05.144917 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.144891    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:19:05.144917 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.144896    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 17 17:19:05.145028 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.144992    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:19:05.145090 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.145051    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:19:05.145231 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.145213    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:19:05.150005 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.149986    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"]
Sep 17 17:19:05.229946 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.229925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/963b1a1f-9276-4463-8c4f-c1d6f44cce27-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.230034 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.229950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.230034 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.229974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.230034 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.229995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/963b1a1f-9276-4463-8c4f-c1d6f44cce27-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.230143 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.230060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tzcr\" (UniqueName: \"kubernetes.io/projected/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kube-api-access-6tzcr\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.330363 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.330337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/963b1a1f-9276-4463-8c4f-c1d6f44cce27-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.330446 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.330368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.330446 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.330393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.330446 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.330424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/963b1a1f-9276-4463-8c4f-c1d6f44cce27-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.330583 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.330456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tzcr\" (UniqueName: \"kubernetes.io/projected/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kube-api-access-6tzcr\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.330583 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:19:05.330504    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 17 17:19:05.330583 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:19:05.330578    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/963b1a1f-9276-4463-8c4f-c1d6f44cce27-proxy-tls podName:963b1a1f-9276-4463-8c4f-c1d6f44cce27 nodeName:}" failed. No retries permitted until 2026-09-17 17:19:05.830559106 +0000 UTC m=+2263.855467387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/963b1a1f-9276-4463-8c4f-c1d6f44cce27-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" (UID: "963b1a1f-9276-4463-8c4f-c1d6f44cce27") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 17 17:19:05.330782 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.330764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.331204 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.331154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/963b1a1f-9276-4463-8c4f-c1d6f44cce27-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.331247 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.331154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.341059 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.341033    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tzcr\" (UniqueName: \"kubernetes.io/projected/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kube-api-access-6tzcr\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.833779 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.833745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/963b1a1f-9276-4463-8c4f-c1d6f44cce27-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:05.836262 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:05.836241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/963b1a1f-9276-4463-8c4f-c1d6f44cce27-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:06.053675 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:06.053632    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:06.327126 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:06.327093    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"]
Sep 17 17:19:06.330841 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:19:06.330809    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963b1a1f_9276_4463_8c4f_c1d6f44cce27.slice/crio-f30031d898a7235f79cf77e2664885aad9d9311eddc779d99267663f5a7738c7 WatchSource:0}: Error finding container f30031d898a7235f79cf77e2664885aad9d9311eddc779d99267663f5a7738c7: Status 404 returned error can't find the container with id f30031d898a7235f79cf77e2664885aad9d9311eddc779d99267663f5a7738c7
Sep 17 17:19:06.446769 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:06.446738    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a825dd20-972f-4f5e-80aa-77db164b9036" path="/var/lib/kubelet/pods/a825dd20-972f-4f5e-80aa-77db164b9036/volumes"
Sep 17 17:19:06.736286 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:06.736197    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" event={"ID":"963b1a1f-9276-4463-8c4f-c1d6f44cce27","Type":"ContainerStarted","Data":"be2253c54894e84c250e9269bcc9d6538700d8182ff4fc76475cf85dbff13d6c"}
Sep 17 17:19:06.736286 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:06.736270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" event={"ID":"963b1a1f-9276-4463-8c4f-c1d6f44cce27","Type":"ContainerStarted","Data":"f30031d898a7235f79cf77e2664885aad9d9311eddc779d99267663f5a7738c7"}
Sep 17 17:19:10.755768 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:10.755696    2577 generic.go:356] "Generic (PLEG): container finished" podID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerID="be2253c54894e84c250e9269bcc9d6538700d8182ff4fc76475cf85dbff13d6c" exitCode=0
Sep 17 17:19:10.756205 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:10.755767    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" event={"ID":"963b1a1f-9276-4463-8c4f-c1d6f44cce27","Type":"ContainerDied","Data":"be2253c54894e84c250e9269bcc9d6538700d8182ff4fc76475cf85dbff13d6c"}
Sep 17 17:19:11.762326 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:11.762291    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" event={"ID":"963b1a1f-9276-4463-8c4f-c1d6f44cce27","Type":"ContainerStarted","Data":"ee19a312722b7709e7773b2fbaaa6e376bb1eacc1841366053b37684991dc19d"}
Sep 17 17:19:11.762803 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:11.762363    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" event={"ID":"963b1a1f-9276-4463-8c4f-c1d6f44cce27","Type":"ContainerStarted","Data":"a7962942a2d8cda865835912b78f928ad4e50fd6636bf8ca22c811c596fb0a6a"}
Sep 17 17:19:11.762803 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:11.762583    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:11.781822 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:11.781777    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podStartSLOduration=2.701123652 podStartE2EDuration="6.781764169s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.080640517s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:19:11.78052725 +0000 UTC m=+2269.805435539" watchObservedRunningTime="2026-09-17 17:19:11.781764169 +0000 UTC m=+2269.806672456"
Sep 17 17:19:12.767090 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:12.767057    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:12.768395 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:12.768363    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 17:19:13.771724 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:13.771680    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 17:19:18.775720 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:18.775692    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:19:18.776271 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:18.776239    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 17:19:28.776626 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:28.776580    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 17:19:38.776306 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:38.776265    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 17:19:48.776964 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:48.776920    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 17:19:58.776902 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:19:58.776861    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 17:20:08.776738 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:08.776697    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 17:20:17.236078 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.235997    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"]
Sep 17 17:20:17.239539 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.239520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.242020 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.242001    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 17 17:20:17.242129 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.242027    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 17 17:20:17.249915 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.249893    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"]
Sep 17 17:20:17.341828 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.341798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/454d6baf-f845-4848-90ef-96eea921ebbb-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xzlr6\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.341986 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.341841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/454d6baf-f845-4848-90ef-96eea921ebbb-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xzlr6\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.341986 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.341880    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/454d6baf-f845-4848-90ef-96eea921ebbb-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xzlr6\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.341986 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.341922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/454d6baf-f845-4848-90ef-96eea921ebbb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xzlr6\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.341986 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.341947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tkg6\" (UniqueName: \"kubernetes.io/projected/454d6baf-f845-4848-90ef-96eea921ebbb-kube-api-access-2tkg6\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xzlr6\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.443236 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.443205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/454d6baf-f845-4848-90ef-96eea921ebbb-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xzlr6\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.443407 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.443261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/454d6baf-f845-4848-90ef-96eea921ebbb-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xzlr6\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.443407 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.443334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/454d6baf-f845-4848-90ef-96eea921ebbb-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xzlr6\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.443407 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.443368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/454d6baf-f845-4848-90ef-96eea921ebbb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xzlr6\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.443407 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.443393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tkg6\" (UniqueName: \"kubernetes.io/projected/454d6baf-f845-4848-90ef-96eea921ebbb-kube-api-access-2tkg6\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xzlr6\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.444097 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.444060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/454d6baf-f845-4848-90ef-96eea921ebbb-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xzlr6\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.444554 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.444525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/454d6baf-f845-4848-90ef-96eea921ebbb-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xzlr6\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.445322 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.445298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/454d6baf-f845-4848-90ef-96eea921ebbb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xzlr6\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.451535 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.451512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/454d6baf-f845-4848-90ef-96eea921ebbb-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xzlr6\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.452639 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.452619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tkg6\" (UniqueName: \"kubernetes.io/projected/454d6baf-f845-4848-90ef-96eea921ebbb-kube-api-access-2tkg6\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xzlr6\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.549895 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.549862    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:17.824362 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:17.824338    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"]
Sep 17 17:20:17.829847 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:20:17.829742    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454d6baf_f845_4848_90ef_96eea921ebbb.slice/crio-d9533573f25d21eed36c31ad17636239553de4d1bfd0e3a52206a617738be21a WatchSource:0}: Error finding container d9533573f25d21eed36c31ad17636239553de4d1bfd0e3a52206a617738be21a: Status 404 returned error can't find the container with id d9533573f25d21eed36c31ad17636239553de4d1bfd0e3a52206a617738be21a
Sep 17 17:20:18.065462 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:18.065425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" event={"ID":"454d6baf-f845-4848-90ef-96eea921ebbb","Type":"ContainerStarted","Data":"114f94649f248900cc36807c5b48813e6eb8d065e677bcd004d6e7b5db5b9fdc"}
Sep 17 17:20:18.065636 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:18.065484    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" event={"ID":"454d6baf-f845-4848-90ef-96eea921ebbb","Type":"ContainerStarted","Data":"d9533573f25d21eed36c31ad17636239553de4d1bfd0e3a52206a617738be21a"}
Sep 17 17:20:18.776662 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:18.776625    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 17:20:22.084723 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:22.084690    2577 generic.go:356] "Generic (PLEG): container finished" podID="454d6baf-f845-4848-90ef-96eea921ebbb" containerID="114f94649f248900cc36807c5b48813e6eb8d065e677bcd004d6e7b5db5b9fdc" exitCode=0
Sep 17 17:20:22.085120 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:22.084756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" event={"ID":"454d6baf-f845-4848-90ef-96eea921ebbb","Type":"ContainerDied","Data":"114f94649f248900cc36807c5b48813e6eb8d065e677bcd004d6e7b5db5b9fdc"}
Sep 17 17:20:22.085881 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:22.085865    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:20:28.776288 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:28.776242    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 17:20:38.442148 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:38.442092    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 17:20:43.189065 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:43.189029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" event={"ID":"454d6baf-f845-4848-90ef-96eea921ebbb","Type":"ContainerStarted","Data":"96acdc93a04d5bc9f42071770fa2d3006ad024863736800536ab056e662702b6"}
Sep 17 17:20:43.189588 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:43.189090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" event={"ID":"454d6baf-f845-4848-90ef-96eea921ebbb","Type":"ContainerStarted","Data":"cd0fcc88cabe697ea31483ad06610daacf77c449878f2b35e0f2901100dd79cc"}
Sep 17 17:20:43.189588 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:43.189362    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:43.189588 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:43.189411    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:43.190617 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:43.190590    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 17:20:43.207137 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:43.207097    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" podStartSLOduration=2.098989884 podStartE2EDuration="26.207083047s" totalImagesPullingTime="20.411791892s" totalInitContainerRuntime="3.696301271s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:20:43.205921679 +0000 UTC m=+2361.230829968" watchObservedRunningTime="2026-09-17 17:20:43.207083047 +0000 UTC m=+2361.231991359"
Sep 17 17:20:44.193411 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:44.193374    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 17:20:48.447548 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:48.447519    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:20:49.197608 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:49.197574    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:20:49.198182 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:49.198136    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 17:20:56.946750 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:56.946715    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"]
Sep 17 17:20:56.947240 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:56.946933    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container" containerID="cri-o://a7962942a2d8cda865835912b78f928ad4e50fd6636bf8ca22c811c596fb0a6a" gracePeriod=30
Sep 17 17:20:56.947240 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:56.946976    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kube-rbac-proxy" containerID="cri-o://ee19a312722b7709e7773b2fbaaa6e376bb1eacc1841366053b37684991dc19d" gracePeriod=30
Sep 17 17:20:57.262086 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:57.261999    2577 generic.go:356] "Generic (PLEG): container finished" podID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerID="ee19a312722b7709e7773b2fbaaa6e376bb1eacc1841366053b37684991dc19d" exitCode=2
Sep 17 17:20:57.262086 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:57.262060    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" event={"ID":"963b1a1f-9276-4463-8c4f-c1d6f44cce27","Type":"ContainerDied","Data":"ee19a312722b7709e7773b2fbaaa6e376bb1eacc1841366053b37684991dc19d"}
Sep 17 17:20:58.442485 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:58.442436    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 17:20:58.772609 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:58.772514    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Sep 17 17:20:59.198454 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:20:59.198415    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 17:21:02.189002 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.188978    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:21:02.224414 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.224381    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tzcr\" (UniqueName: \"kubernetes.io/projected/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kube-api-access-6tzcr\") pod \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") "
Sep 17 17:21:02.224547 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.224435    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/963b1a1f-9276-4463-8c4f-c1d6f44cce27-proxy-tls\") pod \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") "
Sep 17 17:21:02.224547 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.224486    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kserve-provision-location\") pod \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") "
Sep 17 17:21:02.224635 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.224572    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kube-rbac-proxy-sar-config\") pod \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") "
Sep 17 17:21:02.224635 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.224599    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/963b1a1f-9276-4463-8c4f-c1d6f44cce27-cabundle-cert\") pod \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\" (UID: \"963b1a1f-9276-4463-8c4f-c1d6f44cce27\") "
Sep 17 17:21:02.224923 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.224881    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "963b1a1f-9276-4463-8c4f-c1d6f44cce27" (UID: "963b1a1f-9276-4463-8c4f-c1d6f44cce27"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:21:02.225025 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.224913    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "963b1a1f-9276-4463-8c4f-c1d6f44cce27" (UID: "963b1a1f-9276-4463-8c4f-c1d6f44cce27"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:21:02.225025 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.224965    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963b1a1f-9276-4463-8c4f-c1d6f44cce27-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "963b1a1f-9276-4463-8c4f-c1d6f44cce27" (UID: "963b1a1f-9276-4463-8c4f-c1d6f44cce27"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:21:02.226919 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.226894    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963b1a1f-9276-4463-8c4f-c1d6f44cce27-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "963b1a1f-9276-4463-8c4f-c1d6f44cce27" (UID: "963b1a1f-9276-4463-8c4f-c1d6f44cce27"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:21:02.227006 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.226929    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kube-api-access-6tzcr" (OuterVolumeSpecName: "kube-api-access-6tzcr") pod "963b1a1f-9276-4463-8c4f-c1d6f44cce27" (UID: "963b1a1f-9276-4463-8c4f-c1d6f44cce27"). InnerVolumeSpecName "kube-api-access-6tzcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:21:02.286276 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.286249    2577 generic.go:356] "Generic (PLEG): container finished" podID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerID="a7962942a2d8cda865835912b78f928ad4e50fd6636bf8ca22c811c596fb0a6a" exitCode=0
Sep 17 17:21:02.286375 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.286299    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"
Sep 17 17:21:02.286375 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.286324    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" event={"ID":"963b1a1f-9276-4463-8c4f-c1d6f44cce27","Type":"ContainerDied","Data":"a7962942a2d8cda865835912b78f928ad4e50fd6636bf8ca22c811c596fb0a6a"}
Sep 17 17:21:02.286469 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.286406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz" event={"ID":"963b1a1f-9276-4463-8c4f-c1d6f44cce27","Type":"ContainerDied","Data":"f30031d898a7235f79cf77e2664885aad9d9311eddc779d99267663f5a7738c7"}
Sep 17 17:21:02.286469 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.286433    2577 scope.go:117] "RemoveContainer" containerID="ee19a312722b7709e7773b2fbaaa6e376bb1eacc1841366053b37684991dc19d"
Sep 17 17:21:02.294619 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.294604    2577 scope.go:117] "RemoveContainer" containerID="a7962942a2d8cda865835912b78f928ad4e50fd6636bf8ca22c811c596fb0a6a"
Sep 17 17:21:02.301914 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.301896    2577 scope.go:117] "RemoveContainer" containerID="be2253c54894e84c250e9269bcc9d6538700d8182ff4fc76475cf85dbff13d6c"
Sep 17 17:21:02.308669 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.308647    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"]
Sep 17 17:21:02.310670 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.310651    2577 scope.go:117] "RemoveContainer" containerID="ee19a312722b7709e7773b2fbaaa6e376bb1eacc1841366053b37684991dc19d"
Sep 17 17:21:02.310957 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:21:02.310933    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee19a312722b7709e7773b2fbaaa6e376bb1eacc1841366053b37684991dc19d\": container with ID starting with ee19a312722b7709e7773b2fbaaa6e376bb1eacc1841366053b37684991dc19d not found: ID does not exist" containerID="ee19a312722b7709e7773b2fbaaa6e376bb1eacc1841366053b37684991dc19d"
Sep 17 17:21:02.311016 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.310965    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee19a312722b7709e7773b2fbaaa6e376bb1eacc1841366053b37684991dc19d"} err="failed to get container status \"ee19a312722b7709e7773b2fbaaa6e376bb1eacc1841366053b37684991dc19d\": rpc error: code = NotFound desc = could not find container \"ee19a312722b7709e7773b2fbaaa6e376bb1eacc1841366053b37684991dc19d\": container with ID starting with ee19a312722b7709e7773b2fbaaa6e376bb1eacc1841366053b37684991dc19d not found: ID does not exist"
Sep 17 17:21:02.311016 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.310981    2577 scope.go:117] "RemoveContainer" containerID="a7962942a2d8cda865835912b78f928ad4e50fd6636bf8ca22c811c596fb0a6a"
Sep 17 17:21:02.311274 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:21:02.311259    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7962942a2d8cda865835912b78f928ad4e50fd6636bf8ca22c811c596fb0a6a\": container with ID starting with a7962942a2d8cda865835912b78f928ad4e50fd6636bf8ca22c811c596fb0a6a not found: ID does not exist" containerID="a7962942a2d8cda865835912b78f928ad4e50fd6636bf8ca22c811c596fb0a6a"
Sep 17 17:21:02.311334 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.311278    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7962942a2d8cda865835912b78f928ad4e50fd6636bf8ca22c811c596fb0a6a"} err="failed to get container status \"a7962942a2d8cda865835912b78f928ad4e50fd6636bf8ca22c811c596fb0a6a\": rpc error: code = NotFound desc = could not find container \"a7962942a2d8cda865835912b78f928ad4e50fd6636bf8ca22c811c596fb0a6a\": container with ID starting with a7962942a2d8cda865835912b78f928ad4e50fd6636bf8ca22c811c596fb0a6a not found: ID does not exist"
Sep 17 17:21:02.311334 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.311290    2577 scope.go:117] "RemoveContainer" containerID="be2253c54894e84c250e9269bcc9d6538700d8182ff4fc76475cf85dbff13d6c"
Sep 17 17:21:02.311554 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:21:02.311531    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2253c54894e84c250e9269bcc9d6538700d8182ff4fc76475cf85dbff13d6c\": container with ID starting with be2253c54894e84c250e9269bcc9d6538700d8182ff4fc76475cf85dbff13d6c not found: ID does not exist" containerID="be2253c54894e84c250e9269bcc9d6538700d8182ff4fc76475cf85dbff13d6c"
Sep 17 17:21:02.311613 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.311563    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2253c54894e84c250e9269bcc9d6538700d8182ff4fc76475cf85dbff13d6c"} err="failed to get container status \"be2253c54894e84c250e9269bcc9d6538700d8182ff4fc76475cf85dbff13d6c\": rpc error: code = NotFound desc = could not find container \"be2253c54894e84c250e9269bcc9d6538700d8182ff4fc76475cf85dbff13d6c\": container with ID starting with be2253c54894e84c250e9269bcc9d6538700d8182ff4fc76475cf85dbff13d6c not found: ID does not exist"
Sep 17 17:21:02.312195 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.312177    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-qxvfz"]
Sep 17 17:21:02.325099 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.325079    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:21:02.325209 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.325103    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:21:02.325209 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.325119    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/963b1a1f-9276-4463-8c4f-c1d6f44cce27-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:21:02.325209 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.325154    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tzcr\" (UniqueName: \"kubernetes.io/projected/963b1a1f-9276-4463-8c4f-c1d6f44cce27-kube-api-access-6tzcr\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:21:02.325209 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.325184    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/963b1a1f-9276-4463-8c4f-c1d6f44cce27-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:21:02.447318 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:02.447250    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" path="/var/lib/kubelet/pods/963b1a1f-9276-4463-8c4f-c1d6f44cce27/volumes"
Sep 17 17:21:09.198974 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:09.198936    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 17:21:19.198658 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:19.198605    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 17:21:22.585832 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:22.585800    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 17:21:22.590327 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:22.590307    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 17:21:29.198033 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:29.197993    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 17:21:39.198997 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:39.198948    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 17:21:49.198376 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:49.198300    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:21:57.721292 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:57.721260    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"]
Sep 17 17:21:57.721784 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:57.721553    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kserve-container" containerID="cri-o://cd0fcc88cabe697ea31483ad06610daacf77c449878f2b35e0f2901100dd79cc" gracePeriod=30
Sep 17 17:21:57.721784 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:57.721641    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kube-rbac-proxy" containerID="cri-o://96acdc93a04d5bc9f42071770fa2d3006ad024863736800536ab056e662702b6" gracePeriod=30
Sep 17 17:21:58.529027 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:58.528994    2577 generic.go:356] "Generic (PLEG): container finished" podID="454d6baf-f845-4848-90ef-96eea921ebbb" containerID="96acdc93a04d5bc9f42071770fa2d3006ad024863736800536ab056e662702b6" exitCode=2
Sep 17 17:21:58.529212 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:58.529065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" event={"ID":"454d6baf-f845-4848-90ef-96eea921ebbb","Type":"ContainerDied","Data":"96acdc93a04d5bc9f42071770fa2d3006ad024863736800536ab056e662702b6"}
Sep 17 17:21:59.193775 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:59.193736    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 17 17:21:59.198035 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:21:59.198009    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 17:22:01.479412 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.479387    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:22:01.543995 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.543965    2577 generic.go:356] "Generic (PLEG): container finished" podID="454d6baf-f845-4848-90ef-96eea921ebbb" containerID="cd0fcc88cabe697ea31483ad06610daacf77c449878f2b35e0f2901100dd79cc" exitCode=0
Sep 17 17:22:01.544143 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.544018    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"
Sep 17 17:22:01.544143 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.544026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" event={"ID":"454d6baf-f845-4848-90ef-96eea921ebbb","Type":"ContainerDied","Data":"cd0fcc88cabe697ea31483ad06610daacf77c449878f2b35e0f2901100dd79cc"}
Sep 17 17:22:01.544143 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.544097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6" event={"ID":"454d6baf-f845-4848-90ef-96eea921ebbb","Type":"ContainerDied","Data":"d9533573f25d21eed36c31ad17636239553de4d1bfd0e3a52206a617738be21a"}
Sep 17 17:22:01.544143 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.544137    2577 scope.go:117] "RemoveContainer" containerID="96acdc93a04d5bc9f42071770fa2d3006ad024863736800536ab056e662702b6"
Sep 17 17:22:01.544885 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.544868    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/454d6baf-f845-4848-90ef-96eea921ebbb-kube-rbac-proxy-sar-config\") pod \"454d6baf-f845-4848-90ef-96eea921ebbb\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") "
Sep 17 17:22:01.544969 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.544894    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/454d6baf-f845-4848-90ef-96eea921ebbb-proxy-tls\") pod \"454d6baf-f845-4848-90ef-96eea921ebbb\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") "
Sep 17 17:22:01.544969 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.544938    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/454d6baf-f845-4848-90ef-96eea921ebbb-kserve-provision-location\") pod \"454d6baf-f845-4848-90ef-96eea921ebbb\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") "
Sep 17 17:22:01.544969 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.544960    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tkg6\" (UniqueName: \"kubernetes.io/projected/454d6baf-f845-4848-90ef-96eea921ebbb-kube-api-access-2tkg6\") pod \"454d6baf-f845-4848-90ef-96eea921ebbb\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") "
Sep 17 17:22:01.545228 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.545152    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/454d6baf-f845-4848-90ef-96eea921ebbb-cabundle-cert\") pod \"454d6baf-f845-4848-90ef-96eea921ebbb\" (UID: \"454d6baf-f845-4848-90ef-96eea921ebbb\") "
Sep 17 17:22:01.545378 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.545337    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454d6baf-f845-4848-90ef-96eea921ebbb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "454d6baf-f845-4848-90ef-96eea921ebbb" (UID: "454d6baf-f845-4848-90ef-96eea921ebbb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:22:01.545522 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.545411    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454d6baf-f845-4848-90ef-96eea921ebbb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "454d6baf-f845-4848-90ef-96eea921ebbb" (UID: "454d6baf-f845-4848-90ef-96eea921ebbb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:22:01.545522 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.545502    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454d6baf-f845-4848-90ef-96eea921ebbb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "454d6baf-f845-4848-90ef-96eea921ebbb" (UID: "454d6baf-f845-4848-90ef-96eea921ebbb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:22:01.545632 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.545528    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/454d6baf-f845-4848-90ef-96eea921ebbb-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:22:01.545632 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.545551    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/454d6baf-f845-4848-90ef-96eea921ebbb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:22:01.547111 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.547090    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/454d6baf-f845-4848-90ef-96eea921ebbb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "454d6baf-f845-4848-90ef-96eea921ebbb" (UID: "454d6baf-f845-4848-90ef-96eea921ebbb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:22:01.547252 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.547114    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454d6baf-f845-4848-90ef-96eea921ebbb-kube-api-access-2tkg6" (OuterVolumeSpecName: "kube-api-access-2tkg6") pod "454d6baf-f845-4848-90ef-96eea921ebbb" (UID: "454d6baf-f845-4848-90ef-96eea921ebbb"). InnerVolumeSpecName "kube-api-access-2tkg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:22:01.560993 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.560977    2577 scope.go:117] "RemoveContainer" containerID="cd0fcc88cabe697ea31483ad06610daacf77c449878f2b35e0f2901100dd79cc"
Sep 17 17:22:01.567916 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.567898    2577 scope.go:117] "RemoveContainer" containerID="114f94649f248900cc36807c5b48813e6eb8d065e677bcd004d6e7b5db5b9fdc"
Sep 17 17:22:01.574625 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.574607    2577 scope.go:117] "RemoveContainer" containerID="96acdc93a04d5bc9f42071770fa2d3006ad024863736800536ab056e662702b6"
Sep 17 17:22:01.574879 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:22:01.574861    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96acdc93a04d5bc9f42071770fa2d3006ad024863736800536ab056e662702b6\": container with ID starting with 96acdc93a04d5bc9f42071770fa2d3006ad024863736800536ab056e662702b6 not found: ID does not exist" containerID="96acdc93a04d5bc9f42071770fa2d3006ad024863736800536ab056e662702b6"
Sep 17 17:22:01.574926 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.574888    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96acdc93a04d5bc9f42071770fa2d3006ad024863736800536ab056e662702b6"} err="failed to get container status \"96acdc93a04d5bc9f42071770fa2d3006ad024863736800536ab056e662702b6\": rpc error: code = NotFound desc = could not find container \"96acdc93a04d5bc9f42071770fa2d3006ad024863736800536ab056e662702b6\": container with ID starting with 96acdc93a04d5bc9f42071770fa2d3006ad024863736800536ab056e662702b6 not found: ID does not exist"
Sep 17 17:22:01.574926 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.574906    2577 scope.go:117] "RemoveContainer" containerID="cd0fcc88cabe697ea31483ad06610daacf77c449878f2b35e0f2901100dd79cc"
Sep 17 17:22:01.575140 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:22:01.575119    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0fcc88cabe697ea31483ad06610daacf77c449878f2b35e0f2901100dd79cc\": container with ID starting with cd0fcc88cabe697ea31483ad06610daacf77c449878f2b35e0f2901100dd79cc not found: ID does not exist" containerID="cd0fcc88cabe697ea31483ad06610daacf77c449878f2b35e0f2901100dd79cc"
Sep 17 17:22:01.575205 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.575149    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0fcc88cabe697ea31483ad06610daacf77c449878f2b35e0f2901100dd79cc"} err="failed to get container status \"cd0fcc88cabe697ea31483ad06610daacf77c449878f2b35e0f2901100dd79cc\": rpc error: code = NotFound desc = could not find container \"cd0fcc88cabe697ea31483ad06610daacf77c449878f2b35e0f2901100dd79cc\": container with ID starting with cd0fcc88cabe697ea31483ad06610daacf77c449878f2b35e0f2901100dd79cc not found: ID does not exist"
Sep 17 17:22:01.575248 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.575202    2577 scope.go:117] "RemoveContainer" containerID="114f94649f248900cc36807c5b48813e6eb8d065e677bcd004d6e7b5db5b9fdc"
Sep 17 17:22:01.575461 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:22:01.575443    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114f94649f248900cc36807c5b48813e6eb8d065e677bcd004d6e7b5db5b9fdc\": container with ID starting with 114f94649f248900cc36807c5b48813e6eb8d065e677bcd004d6e7b5db5b9fdc not found: ID does not exist" containerID="114f94649f248900cc36807c5b48813e6eb8d065e677bcd004d6e7b5db5b9fdc"
Sep 17 17:22:01.575513 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.575469    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114f94649f248900cc36807c5b48813e6eb8d065e677bcd004d6e7b5db5b9fdc"} err="failed to get container status \"114f94649f248900cc36807c5b48813e6eb8d065e677bcd004d6e7b5db5b9fdc\": rpc error: code = NotFound desc = could not find container \"114f94649f248900cc36807c5b48813e6eb8d065e677bcd004d6e7b5db5b9fdc\": container with ID starting with 114f94649f248900cc36807c5b48813e6eb8d065e677bcd004d6e7b5db5b9fdc not found: ID does not exist"
Sep 17 17:22:01.646434 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.646408    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tkg6\" (UniqueName: \"kubernetes.io/projected/454d6baf-f845-4848-90ef-96eea921ebbb-kube-api-access-2tkg6\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:22:01.646434 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.646429    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/454d6baf-f845-4848-90ef-96eea921ebbb-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:22:01.646577 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.646439    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/454d6baf-f845-4848-90ef-96eea921ebbb-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:22:01.865597 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.865568    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"]
Sep 17 17:22:01.871650 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:01.871626    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xzlr6"]
Sep 17 17:22:02.447747 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:22:02.447713    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" path="/var/lib/kubelet/pods/454d6baf-f845-4848-90ef-96eea921ebbb/volumes"
Sep 17 17:23:08.120404 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.120366    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"]
Sep 17 17:23:08.120916 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.120796    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kserve-container"
Sep 17 17:23:08.120916 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.120816    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kserve-container"
Sep 17 17:23:08.120916 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.120836    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="storage-initializer"
Sep 17 17:23:08.120916 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.120844    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="storage-initializer"
Sep 17 17:23:08.120916 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.120858    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="storage-initializer"
Sep 17 17:23:08.120916 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.120870    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="storage-initializer"
Sep 17 17:23:08.120916 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.120892    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kube-rbac-proxy"
Sep 17 17:23:08.120916 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.120901    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kube-rbac-proxy"
Sep 17 17:23:08.120916 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.120915    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kube-rbac-proxy"
Sep 17 17:23:08.121411 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.120923    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kube-rbac-proxy"
Sep 17 17:23:08.121411 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.120937    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container"
Sep 17 17:23:08.121411 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.120946    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container"
Sep 17 17:23:08.121411 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.121023    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kserve-container"
Sep 17 17:23:08.121411 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.121034    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="963b1a1f-9276-4463-8c4f-c1d6f44cce27" containerName="kube-rbac-proxy"
Sep 17 17:23:08.121411 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.121045    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kserve-container"
Sep 17 17:23:08.121411 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.121055    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="454d6baf-f845-4848-90ef-96eea921ebbb" containerName="kube-rbac-proxy"
Sep 17 17:23:08.124387 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.124366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.126813 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.126788    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 17 17:23:08.127097 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.127080    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 17:23:08.127178 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.127140    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:23:08.127382 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.127367    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:23:08.127438 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.127376    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:23:08.127829 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.127816    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:23:08.127868 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.127822    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:23:08.132638 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.132617    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"]
Sep 17 17:23:08.207084 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.207061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.207214 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.207089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb85cf98-32d4-4c5a-bcb6-77532beea10d-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.207214 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.207108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgh8f\" (UniqueName: \"kubernetes.io/projected/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kube-api-access-tgh8f\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.207214 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.207133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb85cf98-32d4-4c5a-bcb6-77532beea10d-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.207343 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.207219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.308366 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.308331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb85cf98-32d4-4c5a-bcb6-77532beea10d-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.308476 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.308373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgh8f\" (UniqueName: \"kubernetes.io/projected/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kube-api-access-tgh8f\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.308476 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.308411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb85cf98-32d4-4c5a-bcb6-77532beea10d-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.308476 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.308446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.308646 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:23:08.308496    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 17 17:23:08.308646 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.308518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.308646 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:23:08.308569    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb85cf98-32d4-4c5a-bcb6-77532beea10d-proxy-tls podName:fb85cf98-32d4-4c5a-bcb6-77532beea10d nodeName:}" failed. No retries permitted until 2026-09-17 17:23:08.8085499 +0000 UTC m=+2506.833458168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fb85cf98-32d4-4c5a-bcb6-77532beea10d-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-rq8vf" (UID: "fb85cf98-32d4-4c5a-bcb6-77532beea10d") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 17 17:23:08.308897 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.308877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.309049 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.309029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb85cf98-32d4-4c5a-bcb6-77532beea10d-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.309134 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.309115    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.318412 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.318387    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgh8f\" (UniqueName: \"kubernetes.io/projected/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kube-api-access-tgh8f\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.812902 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.812872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb85cf98-32d4-4c5a-bcb6-77532beea10d-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:08.815369 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:08.815346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb85cf98-32d4-4c5a-bcb6-77532beea10d-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rq8vf\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:09.034836 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:09.034802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:09.307207 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:09.307186    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"]
Sep 17 17:23:09.309413 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:23:09.309380    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb85cf98_32d4_4c5a_bcb6_77532beea10d.slice/crio-efdfc4abaf6193fd87f0908d8547e4ee25983dd1b460968307ba87a4570803f1 WatchSource:0}: Error finding container efdfc4abaf6193fd87f0908d8547e4ee25983dd1b460968307ba87a4570803f1: Status 404 returned error can't find the container with id efdfc4abaf6193fd87f0908d8547e4ee25983dd1b460968307ba87a4570803f1
Sep 17 17:23:09.840512 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:09.840476    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" event={"ID":"fb85cf98-32d4-4c5a-bcb6-77532beea10d","Type":"ContainerStarted","Data":"c69331292ff81bfb6a81619cb41ad7fc307c6c1b05fc332011965b632e87a7d7"}
Sep 17 17:23:09.840667 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:09.840546    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" event={"ID":"fb85cf98-32d4-4c5a-bcb6-77532beea10d","Type":"ContainerStarted","Data":"efdfc4abaf6193fd87f0908d8547e4ee25983dd1b460968307ba87a4570803f1"}
Sep 17 17:23:13.860275 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:13.860246    2577 generic.go:356] "Generic (PLEG): container finished" podID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerID="c69331292ff81bfb6a81619cb41ad7fc307c6c1b05fc332011965b632e87a7d7" exitCode=0
Sep 17 17:23:13.860669 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:13.860317    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" event={"ID":"fb85cf98-32d4-4c5a-bcb6-77532beea10d","Type":"ContainerDied","Data":"c69331292ff81bfb6a81619cb41ad7fc307c6c1b05fc332011965b632e87a7d7"}
Sep 17 17:23:14.867217 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:14.867181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" event={"ID":"fb85cf98-32d4-4c5a-bcb6-77532beea10d","Type":"ContainerStarted","Data":"9423bb22cd9d3d53fc5a1764834bc0077d953a4a1faa85051ca6bb3b2a72bc56"}
Sep 17 17:23:14.867613 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:14.867241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" event={"ID":"fb85cf98-32d4-4c5a-bcb6-77532beea10d","Type":"ContainerStarted","Data":"f2c09cc9d2b7096b68c5f9254e336ad57986c77a514225b2b2962d1520ac1b7e"}
Sep 17 17:23:14.867613 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:14.867414    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:14.885077 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:14.885029    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" podStartSLOduration=2.84630401 podStartE2EDuration="6.885015199s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.038711189s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:23:14.883839495 +0000 UTC m=+2512.908747786" watchObservedRunningTime="2026-09-17 17:23:14.885015199 +0000 UTC m=+2512.909923487"
Sep 17 17:23:15.872046 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:15.872011    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:15.873180 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:15.873140    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 17:23:16.876290 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:16.876242    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 17:23:21.880904 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:21.880874    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:23:21.881378 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:21.881350    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 17:23:31.881279 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:31.881237    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 17:23:41.881974 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:41.881889    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 17:23:51.881761 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:23:51.881706    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 17:24:01.881897 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:01.881857    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 17:24:11.881944 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:11.881900    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 17:24:21.882351 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:21.882314    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:24:28.730024 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:28.729992    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"]
Sep 17 17:24:28.730560 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:28.730236    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kserve-container" containerID="cri-o://f2c09cc9d2b7096b68c5f9254e336ad57986c77a514225b2b2962d1520ac1b7e" gracePeriod=30
Sep 17 17:24:28.730560 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:28.730288    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kube-rbac-proxy" containerID="cri-o://9423bb22cd9d3d53fc5a1764834bc0077d953a4a1faa85051ca6bb3b2a72bc56" gracePeriod=30
Sep 17 17:24:29.194905 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:29.194874    2577 generic.go:356] "Generic (PLEG): container finished" podID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerID="9423bb22cd9d3d53fc5a1764834bc0077d953a4a1faa85051ca6bb3b2a72bc56" exitCode=2
Sep 17 17:24:29.195064 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:29.194945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" event={"ID":"fb85cf98-32d4-4c5a-bcb6-77532beea10d","Type":"ContainerDied","Data":"9423bb22cd9d3d53fc5a1764834bc0077d953a4a1faa85051ca6bb3b2a72bc56"}
Sep 17 17:24:31.877327 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:31.877277    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 17 17:24:31.881696 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:31.881663    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 17:24:32.215066 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.215030    2577 generic.go:356] "Generic (PLEG): container finished" podID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerID="f2c09cc9d2b7096b68c5f9254e336ad57986c77a514225b2b2962d1520ac1b7e" exitCode=0
Sep 17 17:24:32.215214 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.215096    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" event={"ID":"fb85cf98-32d4-4c5a-bcb6-77532beea10d","Type":"ContainerDied","Data":"f2c09cc9d2b7096b68c5f9254e336ad57986c77a514225b2b2962d1520ac1b7e"}
Sep 17 17:24:32.271133 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.271109    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:24:32.310152 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.310127    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb85cf98-32d4-4c5a-bcb6-77532beea10d-cabundle-cert\") pod \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") "
Sep 17 17:24:32.310294 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.310208    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kube-rbac-proxy-sar-config\") pod \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") "
Sep 17 17:24:32.310294 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.310238    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgh8f\" (UniqueName: \"kubernetes.io/projected/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kube-api-access-tgh8f\") pod \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") "
Sep 17 17:24:32.310294 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.310260    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb85cf98-32d4-4c5a-bcb6-77532beea10d-proxy-tls\") pod \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") "
Sep 17 17:24:32.310294 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.310282    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kserve-provision-location\") pod \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\" (UID: \"fb85cf98-32d4-4c5a-bcb6-77532beea10d\") "
Sep 17 17:24:32.310621 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.310598    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb85cf98-32d4-4c5a-bcb6-77532beea10d" (UID: "fb85cf98-32d4-4c5a-bcb6-77532beea10d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:24:32.310621 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.310513    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb85cf98-32d4-4c5a-bcb6-77532beea10d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb85cf98-32d4-4c5a-bcb6-77532beea10d" (UID: "fb85cf98-32d4-4c5a-bcb6-77532beea10d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:24:32.310765 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.310644    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb85cf98-32d4-4c5a-bcb6-77532beea10d" (UID: "fb85cf98-32d4-4c5a-bcb6-77532beea10d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:24:32.312298 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.312278    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb85cf98-32d4-4c5a-bcb6-77532beea10d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb85cf98-32d4-4c5a-bcb6-77532beea10d" (UID: "fb85cf98-32d4-4c5a-bcb6-77532beea10d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:24:32.312377 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.312362    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kube-api-access-tgh8f" (OuterVolumeSpecName: "kube-api-access-tgh8f") pod "fb85cf98-32d4-4c5a-bcb6-77532beea10d" (UID: "fb85cf98-32d4-4c5a-bcb6-77532beea10d"). InnerVolumeSpecName "kube-api-access-tgh8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:24:32.410764 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.410712    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:24:32.410764 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.410732    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgh8f\" (UniqueName: \"kubernetes.io/projected/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kube-api-access-tgh8f\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:24:32.410764 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.410742    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb85cf98-32d4-4c5a-bcb6-77532beea10d-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:24:32.410764 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.410751    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb85cf98-32d4-4c5a-bcb6-77532beea10d-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:24:32.410764 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:32.410759    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb85cf98-32d4-4c5a-bcb6-77532beea10d-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:24:33.221371 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.221331    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf" event={"ID":"fb85cf98-32d4-4c5a-bcb6-77532beea10d","Type":"ContainerDied","Data":"efdfc4abaf6193fd87f0908d8547e4ee25983dd1b460968307ba87a4570803f1"}
Sep 17 17:24:33.221371 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.221363    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"
Sep 17 17:24:33.221910 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.221418    2577 scope.go:117] "RemoveContainer" containerID="9423bb22cd9d3d53fc5a1764834bc0077d953a4a1faa85051ca6bb3b2a72bc56"
Sep 17 17:24:33.229933 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.229915    2577 scope.go:117] "RemoveContainer" containerID="f2c09cc9d2b7096b68c5f9254e336ad57986c77a514225b2b2962d1520ac1b7e"
Sep 17 17:24:33.237683 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.237663    2577 scope.go:117] "RemoveContainer" containerID="c69331292ff81bfb6a81619cb41ad7fc307c6c1b05fc332011965b632e87a7d7"
Sep 17 17:24:33.241673 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.241648    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"]
Sep 17 17:24:33.246794 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.246776    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rq8vf"]
Sep 17 17:24:33.940678 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.940645    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"]
Sep 17 17:24:33.940960 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.940948    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="storage-initializer"
Sep 17 17:24:33.941013 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.940962    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="storage-initializer"
Sep 17 17:24:33.941013 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.940979    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kube-rbac-proxy"
Sep 17 17:24:33.941013 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.940984    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kube-rbac-proxy"
Sep 17 17:24:33.941013 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.940993    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kserve-container"
Sep 17 17:24:33.941013 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.940998    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kserve-container"
Sep 17 17:24:33.941190 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.941050    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kube-rbac-proxy"
Sep 17 17:24:33.941190 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.941062    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" containerName="kserve-container"
Sep 17 17:24:33.945517 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.945500    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:33.948003 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.947973    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 17 17:24:33.948127 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.948017    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:24:33.948127 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.948030    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 17:24:33.948127 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.948044    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:24:33.948127 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.947980    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 17 17:24:33.948315 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.948300    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:24:33.953584 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:33.953563    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"]
Sep 17 17:24:34.022485 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:34.022449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:34.022595 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:34.022496    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:34.022595 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:34.022584    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:34.022687 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:34.022631    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cntkh\" (UniqueName: \"kubernetes.io/projected/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kube-api-access-cntkh\") pod \"isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:34.122926 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:34.122902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:34.123033 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:34.122934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:34.123033 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:34.122991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:34.123149 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:34.123029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cntkh\" (UniqueName: \"kubernetes.io/projected/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kube-api-access-cntkh\") pod \"isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:34.123388 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:34.123366    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:34.123602 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:34.123584    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:34.125433 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:34.125412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:34.131104 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:34.131071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cntkh\" (UniqueName: \"kubernetes.io/projected/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kube-api-access-cntkh\") pod \"isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:34.256135 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:34.256079    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:34.447383 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:34.447349    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb85cf98-32d4-4c5a-bcb6-77532beea10d" path="/var/lib/kubelet/pods/fb85cf98-32d4-4c5a-bcb6-77532beea10d/volumes"
Sep 17 17:24:34.529504 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:34.529399    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"]
Sep 17 17:24:34.531663 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:24:34.531639    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee541b81_6ca4_4758_80ce_890a7ee9bc5b.slice/crio-d1744f23f895cd1e61636091113c00c562f7c62ed65f438a30974a17508f59f3 WatchSource:0}: Error finding container d1744f23f895cd1e61636091113c00c562f7c62ed65f438a30974a17508f59f3: Status 404 returned error can't find the container with id d1744f23f895cd1e61636091113c00c562f7c62ed65f438a30974a17508f59f3
Sep 17 17:24:35.240132 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:35.240096    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" event={"ID":"ee541b81-6ca4-4758-80ce-890a7ee9bc5b","Type":"ContainerStarted","Data":"a513d1e39d079b38b0c41174e1b69c56cc01da28f27155b44bb42a0130994db0"}
Sep 17 17:24:35.240309 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:35.240200    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" event={"ID":"ee541b81-6ca4-4758-80ce-890a7ee9bc5b","Type":"ContainerStarted","Data":"d1744f23f895cd1e61636091113c00c562f7c62ed65f438a30974a17508f59f3"}
Sep 17 17:24:36.246404 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:36.246367    2577 generic.go:356] "Generic (PLEG): container finished" podID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerID="a513d1e39d079b38b0c41174e1b69c56cc01da28f27155b44bb42a0130994db0" exitCode=0
Sep 17 17:24:36.246804 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:36.246455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" event={"ID":"ee541b81-6ca4-4758-80ce-890a7ee9bc5b","Type":"ContainerDied","Data":"a513d1e39d079b38b0c41174e1b69c56cc01da28f27155b44bb42a0130994db0"}
Sep 17 17:24:37.253310 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:37.253275    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" event={"ID":"ee541b81-6ca4-4758-80ce-890a7ee9bc5b","Type":"ContainerStarted","Data":"1a0cf97d36ce6bf51ad67bbd0b1af9cf55360b4e55ea672e2dc5f939dfd43c58"}
Sep 17 17:24:37.253689 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:37.253333    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" event={"ID":"ee541b81-6ca4-4758-80ce-890a7ee9bc5b","Type":"ContainerStarted","Data":"91bac5f3f0cffb963c1483fccc488f658ee93c5cece104db391168975585e9da"}
Sep 17 17:24:37.253689 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:37.253374    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:37.253689 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:37.253398    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:37.254593 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:37.254566    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 17:24:37.274696 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:37.274644    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" podStartSLOduration=3.244994478 podStartE2EDuration="4.274628641s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.029634163s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:24:37.27323813 +0000 UTC m=+2595.298146418" watchObservedRunningTime="2026-09-17 17:24:37.274628641 +0000 UTC m=+2595.299536928"
Sep 17 17:24:38.258088 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:38.258046    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 17:24:43.263466 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:43.263435    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:24:43.264005 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:43.263974    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 17:24:53.130932 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.130896    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"]
Sep 17 17:24:53.134522 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.134505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.137328 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.137305    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:24:53.137463 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.137368    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:24:53.137463 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.137374    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 17 17:24:53.137463 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.137415    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 17 17:24:53.142798 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.142773    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"]
Sep 17 17:24:53.144659 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.144638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.144761 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.144667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.144761 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.144687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.144761 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.144751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.144887 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.144770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj7ll\" (UniqueName: \"kubernetes.io/projected/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kube-api-access-gj7ll\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.245281 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.245239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.245410 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.245286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj7ll\" (UniqueName: \"kubernetes.io/projected/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kube-api-access-gj7ll\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.245410 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.245344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.245410 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.245369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.245410 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.245394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.245633 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:24:53.245479    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 17:24:53.245633 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:24:53.245566    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-proxy-tls podName:c4f57ff5-5872-4ef0-8ed1-96518fac4b78 nodeName:}" failed. No retries permitted until 2026-09-17 17:24:53.745546049 +0000 UTC m=+2611.770454324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-cscwr" (UID: "c4f57ff5-5872-4ef0-8ed1-96518fac4b78") : secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 17:24:53.245760 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.245739    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.245923 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.245907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.245961 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.245918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.256137 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.256110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj7ll\" (UniqueName: \"kubernetes.io/projected/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kube-api-access-gj7ll\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.264494 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.264470    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 17:24:53.750063 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.750024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:53.752385 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:53.752358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cscwr\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:54.045713 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:54.045687    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:24:54.327762 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:54.327737    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"]
Sep 17 17:24:54.330227 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:24:54.330196    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f57ff5_5872_4ef0_8ed1_96518fac4b78.slice/crio-ad09fc604a9764d9c701504d92e35dee7212d137ce17a0e6428982c88e4e01cd WatchSource:0}: Error finding container ad09fc604a9764d9c701504d92e35dee7212d137ce17a0e6428982c88e4e01cd: Status 404 returned error can't find the container with id ad09fc604a9764d9c701504d92e35dee7212d137ce17a0e6428982c88e4e01cd
Sep 17 17:24:55.337328 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:55.337292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" event={"ID":"c4f57ff5-5872-4ef0-8ed1-96518fac4b78","Type":"ContainerStarted","Data":"9242e9c0fe8df647a247a5cbafb22d0001379905b66e8cf4234e23dd5515f4f6"}
Sep 17 17:24:55.337804 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:55.337365    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" event={"ID":"c4f57ff5-5872-4ef0-8ed1-96518fac4b78","Type":"ContainerStarted","Data":"ad09fc604a9764d9c701504d92e35dee7212d137ce17a0e6428982c88e4e01cd"}
Sep 17 17:24:59.356710 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:59.356671    2577 generic.go:356] "Generic (PLEG): container finished" podID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerID="9242e9c0fe8df647a247a5cbafb22d0001379905b66e8cf4234e23dd5515f4f6" exitCode=0
Sep 17 17:24:59.357052 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:24:59.356746    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" event={"ID":"c4f57ff5-5872-4ef0-8ed1-96518fac4b78","Type":"ContainerDied","Data":"9242e9c0fe8df647a247a5cbafb22d0001379905b66e8cf4234e23dd5515f4f6"}
Sep 17 17:25:03.264286 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:03.264248    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 17:25:11.422418 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:11.422393    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" event={"ID":"c4f57ff5-5872-4ef0-8ed1-96518fac4b78","Type":"ContainerStarted","Data":"202e2697b25fcc5102dc6dcfd750ad04ec27fdf076508ff02e114994d5efdbb4"}
Sep 17 17:25:11.422712 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:11.422453    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" event={"ID":"c4f57ff5-5872-4ef0-8ed1-96518fac4b78","Type":"ContainerStarted","Data":"e2fed0282701df68577d8f6ebcf1be82bc5e867c3914e77c02777164a8dd9590"}
Sep 17 17:25:11.422712 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:11.422660    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:25:11.440819 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:11.440776    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" podStartSLOduration=1.855853791 podStartE2EDuration="18.440762596s" totalImagesPullingTime="11.944068955s" totalInitContainerRuntime="4.64083985s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:25:11.438859874 +0000 UTC m=+2629.463768175" watchObservedRunningTime="2026-09-17 17:25:11.440762596 +0000 UTC m=+2629.465670882"
Sep 17 17:25:12.427954 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:12.427921    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:25:12.429387 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:12.429354    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 17:25:13.264837 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:13.264795    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 17:25:13.432358 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:13.432321    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 17:25:18.436537 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:18.436508    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:25:18.436985 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:18.436957    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 17:25:23.264713 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:23.264670    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 17:25:28.437872 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:28.437826    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 17:25:33.263898 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:33.263860    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 17:25:38.437045 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:38.437005    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 17:25:43.264385 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:43.264344    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 17:25:48.436912 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:48.436871    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 17:25:53.264853 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:53.264820    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:25:54.026912 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:54.026880    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"]
Sep 17 17:25:54.027128 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:54.027102    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kserve-container" containerID="cri-o://91bac5f3f0cffb963c1483fccc488f658ee93c5cece104db391168975585e9da" gracePeriod=30
Sep 17 17:25:54.027237 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:54.027143    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kube-rbac-proxy" containerID="cri-o://1a0cf97d36ce6bf51ad67bbd0b1af9cf55360b4e55ea672e2dc5f939dfd43c58" gracePeriod=30
Sep 17 17:25:54.610983 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:54.610949    2577 generic.go:356] "Generic (PLEG): container finished" podID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerID="1a0cf97d36ce6bf51ad67bbd0b1af9cf55360b4e55ea672e2dc5f939dfd43c58" exitCode=2
Sep 17 17:25:54.611367 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:54.611023    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" event={"ID":"ee541b81-6ca4-4758-80ce-890a7ee9bc5b","Type":"ContainerDied","Data":"1a0cf97d36ce6bf51ad67bbd0b1af9cf55360b4e55ea672e2dc5f939dfd43c58"}
Sep 17 17:25:58.258951 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.258905    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Sep 17 17:25:58.376005 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.375979    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:25:58.437952 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.437922    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:25:58.445705 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.445683    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cntkh\" (UniqueName: \"kubernetes.io/projected/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kube-api-access-cntkh\") pod \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") "
Sep 17 17:25:58.445830 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.445716    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-proxy-tls\") pod \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") "
Sep 17 17:25:58.445830 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.445754    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kube-rbac-proxy-sar-config\") pod \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") "
Sep 17 17:25:58.445830 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.445804    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kserve-provision-location\") pod \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\" (UID: \"ee541b81-6ca4-4758-80ce-890a7ee9bc5b\") "
Sep 17 17:25:58.446199 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.446152    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ee541b81-6ca4-4758-80ce-890a7ee9bc5b" (UID: "ee541b81-6ca4-4758-80ce-890a7ee9bc5b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:25:58.446199 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.446178    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee541b81-6ca4-4758-80ce-890a7ee9bc5b" (UID: "ee541b81-6ca4-4758-80ce-890a7ee9bc5b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:25:58.447828 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.447801    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ee541b81-6ca4-4758-80ce-890a7ee9bc5b" (UID: "ee541b81-6ca4-4758-80ce-890a7ee9bc5b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:25:58.447909 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.447830    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kube-api-access-cntkh" (OuterVolumeSpecName: "kube-api-access-cntkh") pod "ee541b81-6ca4-4758-80ce-890a7ee9bc5b" (UID: "ee541b81-6ca4-4758-80ce-890a7ee9bc5b"). InnerVolumeSpecName "kube-api-access-cntkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:25:58.546295 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.546269    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:25:58.546414 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.546298    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cntkh\" (UniqueName: \"kubernetes.io/projected/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kube-api-access-cntkh\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:25:58.546414 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.546314    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:25:58.546414 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.546327    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee541b81-6ca4-4758-80ce-890a7ee9bc5b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:25:58.630581 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.630552    2577 generic.go:356] "Generic (PLEG): container finished" podID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerID="91bac5f3f0cffb963c1483fccc488f658ee93c5cece104db391168975585e9da" exitCode=0
Sep 17 17:25:58.630710 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.630606    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"
Sep 17 17:25:58.630710 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.630613    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" event={"ID":"ee541b81-6ca4-4758-80ce-890a7ee9bc5b","Type":"ContainerDied","Data":"91bac5f3f0cffb963c1483fccc488f658ee93c5cece104db391168975585e9da"}
Sep 17 17:25:58.630710 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.630660    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn" event={"ID":"ee541b81-6ca4-4758-80ce-890a7ee9bc5b","Type":"ContainerDied","Data":"d1744f23f895cd1e61636091113c00c562f7c62ed65f438a30974a17508f59f3"}
Sep 17 17:25:58.630710 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.630685    2577 scope.go:117] "RemoveContainer" containerID="1a0cf97d36ce6bf51ad67bbd0b1af9cf55360b4e55ea672e2dc5f939dfd43c58"
Sep 17 17:25:58.639778 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.639757    2577 scope.go:117] "RemoveContainer" containerID="91bac5f3f0cffb963c1483fccc488f658ee93c5cece104db391168975585e9da"
Sep 17 17:25:58.647118 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.647102    2577 scope.go:117] "RemoveContainer" containerID="a513d1e39d079b38b0c41174e1b69c56cc01da28f27155b44bb42a0130994db0"
Sep 17 17:25:58.654689 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.654664    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"]
Sep 17 17:25:58.655125 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.655105    2577 scope.go:117] "RemoveContainer" containerID="1a0cf97d36ce6bf51ad67bbd0b1af9cf55360b4e55ea672e2dc5f939dfd43c58"
Sep 17 17:25:58.655551 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:25:58.655523    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a0cf97d36ce6bf51ad67bbd0b1af9cf55360b4e55ea672e2dc5f939dfd43c58\": container with ID starting with 1a0cf97d36ce6bf51ad67bbd0b1af9cf55360b4e55ea672e2dc5f939dfd43c58 not found: ID does not exist" containerID="1a0cf97d36ce6bf51ad67bbd0b1af9cf55360b4e55ea672e2dc5f939dfd43c58"
Sep 17 17:25:58.655669 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.655579    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a0cf97d36ce6bf51ad67bbd0b1af9cf55360b4e55ea672e2dc5f939dfd43c58"} err="failed to get container status \"1a0cf97d36ce6bf51ad67bbd0b1af9cf55360b4e55ea672e2dc5f939dfd43c58\": rpc error: code = NotFound desc = could not find container \"1a0cf97d36ce6bf51ad67bbd0b1af9cf55360b4e55ea672e2dc5f939dfd43c58\": container with ID starting with 1a0cf97d36ce6bf51ad67bbd0b1af9cf55360b4e55ea672e2dc5f939dfd43c58 not found: ID does not exist"
Sep 17 17:25:58.655669 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.655618    2577 scope.go:117] "RemoveContainer" containerID="91bac5f3f0cffb963c1483fccc488f658ee93c5cece104db391168975585e9da"
Sep 17 17:25:58.655918 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:25:58.655890    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91bac5f3f0cffb963c1483fccc488f658ee93c5cece104db391168975585e9da\": container with ID starting with 91bac5f3f0cffb963c1483fccc488f658ee93c5cece104db391168975585e9da not found: ID does not exist" containerID="91bac5f3f0cffb963c1483fccc488f658ee93c5cece104db391168975585e9da"
Sep 17 17:25:58.656063 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.655924    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bac5f3f0cffb963c1483fccc488f658ee93c5cece104db391168975585e9da"} err="failed to get container status \"91bac5f3f0cffb963c1483fccc488f658ee93c5cece104db391168975585e9da\": rpc error: code = NotFound desc = could not find container \"91bac5f3f0cffb963c1483fccc488f658ee93c5cece104db391168975585e9da\": container with ID starting with 91bac5f3f0cffb963c1483fccc488f658ee93c5cece104db391168975585e9da not found: ID does not exist"
Sep 17 17:25:58.656063 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.655946    2577 scope.go:117] "RemoveContainer" containerID="a513d1e39d079b38b0c41174e1b69c56cc01da28f27155b44bb42a0130994db0"
Sep 17 17:25:58.656306 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:25:58.656284    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a513d1e39d079b38b0c41174e1b69c56cc01da28f27155b44bb42a0130994db0\": container with ID starting with a513d1e39d079b38b0c41174e1b69c56cc01da28f27155b44bb42a0130994db0 not found: ID does not exist" containerID="a513d1e39d079b38b0c41174e1b69c56cc01da28f27155b44bb42a0130994db0"
Sep 17 17:25:58.656369 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.656313    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a513d1e39d079b38b0c41174e1b69c56cc01da28f27155b44bb42a0130994db0"} err="failed to get container status \"a513d1e39d079b38b0c41174e1b69c56cc01da28f27155b44bb42a0130994db0\": rpc error: code = NotFound desc = could not find container \"a513d1e39d079b38b0c41174e1b69c56cc01da28f27155b44bb42a0130994db0\": container with ID starting with a513d1e39d079b38b0c41174e1b69c56cc01da28f27155b44bb42a0130994db0 not found: ID does not exist"
Sep 17 17:25:58.657481 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:25:58.657462    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7fb7758f7c-7vjqn"]
Sep 17 17:26:00.447941 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:00.447908    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" path="/var/lib/kubelet/pods/ee541b81-6ca4-4758-80ce-890a7ee9bc5b/volumes"
Sep 17 17:26:08.535588 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:08.535557    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"]
Sep 17 17:26:08.536018 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:08.535767    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="kserve-container" containerID="cri-o://e2fed0282701df68577d8f6ebcf1be82bc5e867c3914e77c02777164a8dd9590" gracePeriod=30
Sep 17 17:26:08.536018 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:08.535822    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="kube-rbac-proxy" containerID="cri-o://202e2697b25fcc5102dc6dcfd750ad04ec27fdf076508ff02e114994d5efdbb4" gracePeriod=30
Sep 17 17:26:08.683844 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:08.683817    2577 generic.go:356] "Generic (PLEG): container finished" podID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerID="202e2697b25fcc5102dc6dcfd750ad04ec27fdf076508ff02e114994d5efdbb4" exitCode=2
Sep 17 17:26:08.683944 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:08.683856    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" event={"ID":"c4f57ff5-5872-4ef0-8ed1-96518fac4b78","Type":"ContainerDied","Data":"202e2697b25fcc5102dc6dcfd750ad04ec27fdf076508ff02e114994d5efdbb4"}
Sep 17 17:26:11.180074 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.180046    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:26:11.238270 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.238247    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-proxy-tls\") pod \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") "
Sep 17 17:26:11.238396 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.238277    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj7ll\" (UniqueName: \"kubernetes.io/projected/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kube-api-access-gj7ll\") pod \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") "
Sep 17 17:26:11.238396 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.238302    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kube-rbac-proxy-sar-config\") pod \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") "
Sep 17 17:26:11.238396 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.238353    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kserve-provision-location\") pod \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") "
Sep 17 17:26:11.238396 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.238393    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-cabundle-cert\") pod \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\" (UID: \"c4f57ff5-5872-4ef0-8ed1-96518fac4b78\") "
Sep 17 17:26:11.238765 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.238739    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c4f57ff5-5872-4ef0-8ed1-96518fac4b78" (UID: "c4f57ff5-5872-4ef0-8ed1-96518fac4b78"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:26:11.238874 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.238828    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c4f57ff5-5872-4ef0-8ed1-96518fac4b78" (UID: "c4f57ff5-5872-4ef0-8ed1-96518fac4b78"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:26:11.240355 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.240332    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kube-api-access-gj7ll" (OuterVolumeSpecName: "kube-api-access-gj7ll") pod "c4f57ff5-5872-4ef0-8ed1-96518fac4b78" (UID: "c4f57ff5-5872-4ef0-8ed1-96518fac4b78"). InnerVolumeSpecName "kube-api-access-gj7ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:26:11.240568 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.240550    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c4f57ff5-5872-4ef0-8ed1-96518fac4b78" (UID: "c4f57ff5-5872-4ef0-8ed1-96518fac4b78"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:26:11.246717 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.246687    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c4f57ff5-5872-4ef0-8ed1-96518fac4b78" (UID: "c4f57ff5-5872-4ef0-8ed1-96518fac4b78"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:26:11.339677 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.339646    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:26:11.339677 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.339678    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:26:11.339807 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.339689    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:26:11.339807 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.339697    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj7ll\" (UniqueName: \"kubernetes.io/projected/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kube-api-access-gj7ll\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:26:11.339807 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.339706    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4f57ff5-5872-4ef0-8ed1-96518fac4b78-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:26:11.698710 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.698630    2577 generic.go:356] "Generic (PLEG): container finished" podID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerID="e2fed0282701df68577d8f6ebcf1be82bc5e867c3914e77c02777164a8dd9590" exitCode=0
Sep 17 17:26:11.698710 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.698686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" event={"ID":"c4f57ff5-5872-4ef0-8ed1-96518fac4b78","Type":"ContainerDied","Data":"e2fed0282701df68577d8f6ebcf1be82bc5e867c3914e77c02777164a8dd9590"}
Sep 17 17:26:11.698710 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.698701    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"
Sep 17 17:26:11.698948 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.698732    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr" event={"ID":"c4f57ff5-5872-4ef0-8ed1-96518fac4b78","Type":"ContainerDied","Data":"ad09fc604a9764d9c701504d92e35dee7212d137ce17a0e6428982c88e4e01cd"}
Sep 17 17:26:11.698948 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.698756    2577 scope.go:117] "RemoveContainer" containerID="202e2697b25fcc5102dc6dcfd750ad04ec27fdf076508ff02e114994d5efdbb4"
Sep 17 17:26:11.708082 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.708066    2577 scope.go:117] "RemoveContainer" containerID="e2fed0282701df68577d8f6ebcf1be82bc5e867c3914e77c02777164a8dd9590"
Sep 17 17:26:11.715565 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.715537    2577 scope.go:117] "RemoveContainer" containerID="9242e9c0fe8df647a247a5cbafb22d0001379905b66e8cf4234e23dd5515f4f6"
Sep 17 17:26:11.721908 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.721880    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"]
Sep 17 17:26:11.723753 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.723727    2577 scope.go:117] "RemoveContainer" containerID="202e2697b25fcc5102dc6dcfd750ad04ec27fdf076508ff02e114994d5efdbb4"
Sep 17 17:26:11.724245 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:26:11.724217    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"202e2697b25fcc5102dc6dcfd750ad04ec27fdf076508ff02e114994d5efdbb4\": container with ID starting with 202e2697b25fcc5102dc6dcfd750ad04ec27fdf076508ff02e114994d5efdbb4 not found: ID does not exist" containerID="202e2697b25fcc5102dc6dcfd750ad04ec27fdf076508ff02e114994d5efdbb4"
Sep 17 17:26:11.724354 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.724283    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"202e2697b25fcc5102dc6dcfd750ad04ec27fdf076508ff02e114994d5efdbb4"} err="failed to get container status \"202e2697b25fcc5102dc6dcfd750ad04ec27fdf076508ff02e114994d5efdbb4\": rpc error: code = NotFound desc = could not find container \"202e2697b25fcc5102dc6dcfd750ad04ec27fdf076508ff02e114994d5efdbb4\": container with ID starting with 202e2697b25fcc5102dc6dcfd750ad04ec27fdf076508ff02e114994d5efdbb4 not found: ID does not exist"
Sep 17 17:26:11.724354 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.724304    2577 scope.go:117] "RemoveContainer" containerID="e2fed0282701df68577d8f6ebcf1be82bc5e867c3914e77c02777164a8dd9590"
Sep 17 17:26:11.724583 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:26:11.724557    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2fed0282701df68577d8f6ebcf1be82bc5e867c3914e77c02777164a8dd9590\": container with ID starting with e2fed0282701df68577d8f6ebcf1be82bc5e867c3914e77c02777164a8dd9590 not found: ID does not exist" containerID="e2fed0282701df68577d8f6ebcf1be82bc5e867c3914e77c02777164a8dd9590"
Sep 17 17:26:11.724639 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.724594    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2fed0282701df68577d8f6ebcf1be82bc5e867c3914e77c02777164a8dd9590"} err="failed to get container status \"e2fed0282701df68577d8f6ebcf1be82bc5e867c3914e77c02777164a8dd9590\": rpc error: code = NotFound desc = could not find container \"e2fed0282701df68577d8f6ebcf1be82bc5e867c3914e77c02777164a8dd9590\": container with ID starting with e2fed0282701df68577d8f6ebcf1be82bc5e867c3914e77c02777164a8dd9590 not found: ID does not exist"
Sep 17 17:26:11.724639 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.724617    2577 scope.go:117] "RemoveContainer" containerID="9242e9c0fe8df647a247a5cbafb22d0001379905b66e8cf4234e23dd5515f4f6"
Sep 17 17:26:11.724914 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:26:11.724896    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9242e9c0fe8df647a247a5cbafb22d0001379905b66e8cf4234e23dd5515f4f6\": container with ID starting with 9242e9c0fe8df647a247a5cbafb22d0001379905b66e8cf4234e23dd5515f4f6 not found: ID does not exist" containerID="9242e9c0fe8df647a247a5cbafb22d0001379905b66e8cf4234e23dd5515f4f6"
Sep 17 17:26:11.724977 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.724922    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9242e9c0fe8df647a247a5cbafb22d0001379905b66e8cf4234e23dd5515f4f6"} err="failed to get container status \"9242e9c0fe8df647a247a5cbafb22d0001379905b66e8cf4234e23dd5515f4f6\": rpc error: code = NotFound desc = could not find container \"9242e9c0fe8df647a247a5cbafb22d0001379905b66e8cf4234e23dd5515f4f6\": container with ID starting with 9242e9c0fe8df647a247a5cbafb22d0001379905b66e8cf4234e23dd5515f4f6 not found: ID does not exist"
Sep 17 17:26:11.725444 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:11.725429    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cscwr"]
Sep 17 17:26:12.447093 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:12.447056    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" path="/var/lib/kubelet/pods/c4f57ff5-5872-4ef0-8ed1-96518fac4b78/volumes"
Sep 17 17:26:13.722413 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.722376    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"]
Sep 17 17:26:13.722880 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.722830    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="storage-initializer"
Sep 17 17:26:13.722880 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.722849    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="storage-initializer"
Sep 17 17:26:13.723003 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.722865    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kube-rbac-proxy"
Sep 17 17:26:13.723003 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.722895    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kube-rbac-proxy"
Sep 17 17:26:13.723003 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.722910    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="storage-initializer"
Sep 17 17:26:13.723003 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.722920    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="storage-initializer"
Sep 17 17:26:13.723003 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.722945    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kserve-container"
Sep 17 17:26:13.723003 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.722953    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kserve-container"
Sep 17 17:26:13.723003 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.722963    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="kube-rbac-proxy"
Sep 17 17:26:13.723003 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.722972    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="kube-rbac-proxy"
Sep 17 17:26:13.723003 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.722983    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="kserve-container"
Sep 17 17:26:13.723003 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.722992    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="kserve-container"
Sep 17 17:26:13.723547 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.723080    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="kube-rbac-proxy"
Sep 17 17:26:13.723547 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.723092    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kserve-container"
Sep 17 17:26:13.723547 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.723101    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee541b81-6ca4-4758-80ce-890a7ee9bc5b" containerName="kube-rbac-proxy"
Sep 17 17:26:13.723547 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.723112    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f57ff5-5872-4ef0-8ed1-96518fac4b78" containerName="kserve-container"
Sep 17 17:26:13.729913 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.728862    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:13.732246 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.732219    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 17 17:26:13.732650 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.732630    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v89r4\""
Sep 17 17:26:13.732650 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.732639    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:26:13.732843 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.732673    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:26:13.732843 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.732720    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 17 17:26:13.732843 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.732731    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:26:13.732843 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.732678    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:26:13.737338 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.737317    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"]
Sep 17 17:26:13.756301 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.756275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncg6m\" (UniqueName: \"kubernetes.io/projected/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kube-api-access-ncg6m\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:13.756405 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.756312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:13.756405 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.756338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:13.756532 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.756416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4e9adfb-a985-4062-a88e-d34630ef9bf0-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:13.756532 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.756477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4e9adfb-a985-4062-a88e-d34630ef9bf0-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:13.856879 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.856835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4e9adfb-a985-4062-a88e-d34630ef9bf0-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:13.856879 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.856884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncg6m\" (UniqueName: \"kubernetes.io/projected/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kube-api-access-ncg6m\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:13.857043 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.856918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:13.857043 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.856946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:13.857043 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.856975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4e9adfb-a985-4062-a88e-d34630ef9bf0-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:13.857043 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:26:13.856987    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 17 17:26:13.857328 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:26:13.857057    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4e9adfb-a985-4062-a88e-d34630ef9bf0-proxy-tls podName:a4e9adfb-a985-4062-a88e-d34630ef9bf0 nodeName:}" failed. No retries permitted until 2026-09-17 17:26:14.35703362 +0000 UTC m=+2692.381941891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a4e9adfb-a985-4062-a88e-d34630ef9bf0-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-tn8k8" (UID: "a4e9adfb-a985-4062-a88e-d34630ef9bf0") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 17 17:26:13.857401 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.857343    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:13.857642 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.857625    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4e9adfb-a985-4062-a88e-d34630ef9bf0-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:13.857681 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.857637    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:13.868377 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:13.868357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncg6m\" (UniqueName: \"kubernetes.io/projected/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kube-api-access-ncg6m\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:14.361480 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:14.361443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4e9adfb-a985-4062-a88e-d34630ef9bf0-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:14.363789 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:14.363770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4e9adfb-a985-4062-a88e-d34630ef9bf0-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tn8k8\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:14.640246 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:14.640183    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:14.911986 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:14.911962    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"]
Sep 17 17:26:14.914179 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:26:14.914135    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e9adfb_a985_4062_a88e_d34630ef9bf0.slice/crio-a1e49650c630a77f4cb1f8180f66f3863ee30a2f3d2d44bb1e44bfed573f0016 WatchSource:0}: Error finding container a1e49650c630a77f4cb1f8180f66f3863ee30a2f3d2d44bb1e44bfed573f0016: Status 404 returned error can't find the container with id a1e49650c630a77f4cb1f8180f66f3863ee30a2f3d2d44bb1e44bfed573f0016
Sep 17 17:26:15.720104 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:15.720065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" event={"ID":"a4e9adfb-a985-4062-a88e-d34630ef9bf0","Type":"ContainerStarted","Data":"3905f045ea4a065578558fb5058a30d9af50f65e26bfe0592bec8da68c9bf6eb"}
Sep 17 17:26:15.720308 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:15.720121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" event={"ID":"a4e9adfb-a985-4062-a88e-d34630ef9bf0","Type":"ContainerStarted","Data":"a1e49650c630a77f4cb1f8180f66f3863ee30a2f3d2d44bb1e44bfed573f0016"}
Sep 17 17:26:19.739786 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:19.739756    2577 generic.go:356] "Generic (PLEG): container finished" podID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerID="3905f045ea4a065578558fb5058a30d9af50f65e26bfe0592bec8da68c9bf6eb" exitCode=0
Sep 17 17:26:19.740202 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:19.739831    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" event={"ID":"a4e9adfb-a985-4062-a88e-d34630ef9bf0","Type":"ContainerDied","Data":"3905f045ea4a065578558fb5058a30d9af50f65e26bfe0592bec8da68c9bf6eb"}
Sep 17 17:26:20.746557 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:20.746521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" event={"ID":"a4e9adfb-a985-4062-a88e-d34630ef9bf0","Type":"ContainerStarted","Data":"f44eb933eec3d8bdf50c3faa5e4f0996d1bf3e49fe6f95279f8ffb5c88d231a1"}
Sep 17 17:26:20.747023 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:20.746591    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" event={"ID":"a4e9adfb-a985-4062-a88e-d34630ef9bf0","Type":"ContainerStarted","Data":"ba97a661d66dc6269cbb09668ce7fb98f5ea783f1444f25aff4b7adf625d7e55"}
Sep 17 17:26:20.747023 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:20.746805    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:20.765622 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:20.765582    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" podStartSLOduration=3.510162702 podStartE2EDuration="7.765569797s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.255407095s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:26:20.764266796 +0000 UTC m=+2698.789175084" watchObservedRunningTime="2026-09-17 17:26:20.765569797 +0000 UTC m=+2698.790478084"
Sep 17 17:26:21.751508 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:21.751474    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:21.752458 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:21.752428    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 17:26:22.614621 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:22.614596    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 17:26:22.619860 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:22.619838    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 17:26:22.755640 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:22.755603    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 17:26:27.760572 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:27.760538    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:26:27.761050 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:27.761024    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 17:26:37.761728 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:37.761686    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 17:26:47.761381 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:47.761335    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 17:26:57.761376 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:26:57.761333    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 17:27:07.762402 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:07.762370    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:27:15.266856 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.266825    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"]
Sep 17 17:27:15.267302 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.267058    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kserve-container" containerID="cri-o://ba97a661d66dc6269cbb09668ce7fb98f5ea783f1444f25aff4b7adf625d7e55" gracePeriod=30
Sep 17 17:27:15.267302 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.267090    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kube-rbac-proxy" containerID="cri-o://f44eb933eec3d8bdf50c3faa5e4f0996d1bf3e49fe6f95279f8ffb5c88d231a1" gracePeriod=30
Sep 17 17:27:15.290110 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.290087    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"]
Sep 17 17:27:15.293527 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.293512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:15.296053 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.296025    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 17 17:27:15.296147 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.296075    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 17:27:15.296147 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.296095    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 17 17:27:15.304306 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.304286    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"]
Sep 17 17:27:15.408083 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.408059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/239ad695-2889-426d-b199-470808c290b3-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:15.408209 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.408186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/239ad695-2889-426d-b199-470808c290b3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:15.408256 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.408230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/239ad695-2889-426d-b199-470808c290b3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:15.408293 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.408258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzbzn\" (UniqueName: \"kubernetes.io/projected/239ad695-2889-426d-b199-470808c290b3-kube-api-access-kzbzn\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:15.509459 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.509422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/239ad695-2889-426d-b199-470808c290b3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:15.509588 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.509472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/239ad695-2889-426d-b199-470808c290b3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:15.509588 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.509502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzbzn\" (UniqueName: \"kubernetes.io/projected/239ad695-2889-426d-b199-470808c290b3-kube-api-access-kzbzn\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:15.509588 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.509536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/239ad695-2889-426d-b199-470808c290b3-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:15.509823 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.509800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/239ad695-2889-426d-b199-470808c290b3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:15.510396 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.510373    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/239ad695-2889-426d-b199-470808c290b3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:15.511821 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.511799    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/239ad695-2889-426d-b199-470808c290b3-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:15.517734 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.517664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzbzn\" (UniqueName: \"kubernetes.io/projected/239ad695-2889-426d-b199-470808c290b3-kube-api-access-kzbzn\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:15.603917 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.603895    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:15.880388 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.880363    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"]
Sep 17 17:27:15.882857 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:27:15.882831    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239ad695_2889_426d_b199_470808c290b3.slice/crio-2bb20eab7ceb2f5ab9aa78f0b4e433b0abd6779a82dced3559d7da65dbef8aa8 WatchSource:0}: Error finding container 2bb20eab7ceb2f5ab9aa78f0b4e433b0abd6779a82dced3559d7da65dbef8aa8: Status 404 returned error can't find the container with id 2bb20eab7ceb2f5ab9aa78f0b4e433b0abd6779a82dced3559d7da65dbef8aa8
Sep 17 17:27:15.985361 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.985331    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc" event={"ID":"239ad695-2889-426d-b199-470808c290b3","Type":"ContainerStarted","Data":"4c590c5df80cc0057eaba3252bc3521843f1edbc74a40a265bda699f0bb9858f"}
Sep 17 17:27:15.985485 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.985398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc" event={"ID":"239ad695-2889-426d-b199-470808c290b3","Type":"ContainerStarted","Data":"2bb20eab7ceb2f5ab9aa78f0b4e433b0abd6779a82dced3559d7da65dbef8aa8"}
Sep 17 17:27:15.987608 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.987585    2577 generic.go:356] "Generic (PLEG): container finished" podID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerID="f44eb933eec3d8bdf50c3faa5e4f0996d1bf3e49fe6f95279f8ffb5c88d231a1" exitCode=2
Sep 17 17:27:15.987702 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:15.987636    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" event={"ID":"a4e9adfb-a985-4062-a88e-d34630ef9bf0","Type":"ContainerDied","Data":"f44eb933eec3d8bdf50c3faa5e4f0996d1bf3e49fe6f95279f8ffb5c88d231a1"}
Sep 17 17:27:17.755843 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:17.755800    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Sep 17 17:27:17.761102 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:17.761070    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 17:27:17.999146 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:17.999121    2577 generic.go:356] "Generic (PLEG): container finished" podID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerID="ba97a661d66dc6269cbb09668ce7fb98f5ea783f1444f25aff4b7adf625d7e55" exitCode=0
Sep 17 17:27:17.999272 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:17.999199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" event={"ID":"a4e9adfb-a985-4062-a88e-d34630ef9bf0","Type":"ContainerDied","Data":"ba97a661d66dc6269cbb09668ce7fb98f5ea783f1444f25aff4b7adf625d7e55"}
Sep 17 17:27:18.112278 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.112255    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:27:18.235533 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.235508    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4e9adfb-a985-4062-a88e-d34630ef9bf0-cabundle-cert\") pod \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") "
Sep 17 17:27:18.235649 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.235547    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kube-rbac-proxy-sar-config\") pod \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") "
Sep 17 17:27:18.235649 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.235576    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncg6m\" (UniqueName: \"kubernetes.io/projected/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kube-api-access-ncg6m\") pod \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") "
Sep 17 17:27:18.235649 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.235599    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4e9adfb-a985-4062-a88e-d34630ef9bf0-proxy-tls\") pod \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") "
Sep 17 17:27:18.235649 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.235642    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kserve-provision-location\") pod \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\" (UID: \"a4e9adfb-a985-4062-a88e-d34630ef9bf0\") "
Sep 17 17:27:18.236011 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.235923    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e9adfb-a985-4062-a88e-d34630ef9bf0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a4e9adfb-a985-4062-a88e-d34630ef9bf0" (UID: "a4e9adfb-a985-4062-a88e-d34630ef9bf0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:27:18.236081 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.236053    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a4e9adfb-a985-4062-a88e-d34630ef9bf0" (UID: "a4e9adfb-a985-4062-a88e-d34630ef9bf0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:27:18.237714 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.237691    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kube-api-access-ncg6m" (OuterVolumeSpecName: "kube-api-access-ncg6m") pod "a4e9adfb-a985-4062-a88e-d34630ef9bf0" (UID: "a4e9adfb-a985-4062-a88e-d34630ef9bf0"). InnerVolumeSpecName "kube-api-access-ncg6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:27:18.237812 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.237741    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4e9adfb-a985-4062-a88e-d34630ef9bf0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4e9adfb-a985-4062-a88e-d34630ef9bf0" (UID: "a4e9adfb-a985-4062-a88e-d34630ef9bf0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:27:18.245623 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.245600    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4e9adfb-a985-4062-a88e-d34630ef9bf0" (UID: "a4e9adfb-a985-4062-a88e-d34630ef9bf0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:27:18.336366 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.336332    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4e9adfb-a985-4062-a88e-d34630ef9bf0-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:27:18.336366 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.336366    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:27:18.336505 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.336381    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncg6m\" (UniqueName: \"kubernetes.io/projected/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kube-api-access-ncg6m\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:27:18.336505 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.336394    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4e9adfb-a985-4062-a88e-d34630ef9bf0-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:27:18.336505 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:18.336407    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4e9adfb-a985-4062-a88e-d34630ef9bf0-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:27:19.005384 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:19.005354    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"
Sep 17 17:27:19.005965 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:19.005377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8" event={"ID":"a4e9adfb-a985-4062-a88e-d34630ef9bf0","Type":"ContainerDied","Data":"a1e49650c630a77f4cb1f8180f66f3863ee30a2f3d2d44bb1e44bfed573f0016"}
Sep 17 17:27:19.005965 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:19.005447    2577 scope.go:117] "RemoveContainer" containerID="f44eb933eec3d8bdf50c3faa5e4f0996d1bf3e49fe6f95279f8ffb5c88d231a1"
Sep 17 17:27:19.015786 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:19.015763    2577 scope.go:117] "RemoveContainer" containerID="ba97a661d66dc6269cbb09668ce7fb98f5ea783f1444f25aff4b7adf625d7e55"
Sep 17 17:27:19.023495 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:19.023478    2577 scope.go:117] "RemoveContainer" containerID="3905f045ea4a065578558fb5058a30d9af50f65e26bfe0592bec8da68c9bf6eb"
Sep 17 17:27:19.024570 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:19.024550    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"]
Sep 17 17:27:19.028354 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:19.028331    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tn8k8"]
Sep 17 17:27:20.011019 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.010992    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc_239ad695-2889-426d-b199-470808c290b3/storage-initializer/0.log"
Sep 17 17:27:20.011501 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.011027    2577 generic.go:356] "Generic (PLEG): container finished" podID="239ad695-2889-426d-b199-470808c290b3" containerID="4c590c5df80cc0057eaba3252bc3521843f1edbc74a40a265bda699f0bb9858f" exitCode=1
Sep 17 17:27:20.011501 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.011107    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc" event={"ID":"239ad695-2889-426d-b199-470808c290b3","Type":"ContainerDied","Data":"4c590c5df80cc0057eaba3252bc3521843f1edbc74a40a265bda699f0bb9858f"}
Sep 17 17:27:20.406799 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.406767    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"]
Sep 17 17:27:20.407099 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.407087    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kserve-container"
Sep 17 17:27:20.407145 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.407101    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kserve-container"
Sep 17 17:27:20.407145 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.407110    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="storage-initializer"
Sep 17 17:27:20.407145 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.407116    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="storage-initializer"
Sep 17 17:27:20.407145 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.407130    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kube-rbac-proxy"
Sep 17 17:27:20.407145 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.407136    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kube-rbac-proxy"
Sep 17 17:27:20.407316 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.407206    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kube-rbac-proxy"
Sep 17 17:27:20.407316 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.407214    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" containerName="kserve-container"
Sep 17 17:27:20.410305 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.410287    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.412825 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.412798    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 17 17:27:20.412825 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.412812    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 17:27:20.413192 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.412812    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:27:20.413192 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.412866    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:27:20.420958 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.420937    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"]
Sep 17 17:27:20.447429 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.447397    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4e9adfb-a985-4062-a88e-d34630ef9bf0" path="/var/lib/kubelet/pods/a4e9adfb-a985-4062-a88e-d34630ef9bf0/volumes"
Sep 17 17:27:20.552697 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.552670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4cnh\" (UniqueName: \"kubernetes.io/projected/f92d6e24-f969-4434-82fc-a5a7726adf0c-kube-api-access-n4cnh\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.552807 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.552704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f92d6e24-f969-4434-82fc-a5a7726adf0c-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.552807 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.552781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f92d6e24-f969-4434-82fc-a5a7726adf0c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.552883 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.552816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f92d6e24-f969-4434-82fc-a5a7726adf0c-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.552883 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.552848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f92d6e24-f969-4434-82fc-a5a7726adf0c-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.653515 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.653483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f92d6e24-f969-4434-82fc-a5a7726adf0c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.653654 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.653518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f92d6e24-f969-4434-82fc-a5a7726adf0c-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.653654 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.653545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f92d6e24-f969-4434-82fc-a5a7726adf0c-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.653755 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.653660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4cnh\" (UniqueName: \"kubernetes.io/projected/f92d6e24-f969-4434-82fc-a5a7726adf0c-kube-api-access-n4cnh\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.653755 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.653713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f92d6e24-f969-4434-82fc-a5a7726adf0c-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.653890 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.653870    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f92d6e24-f969-4434-82fc-a5a7726adf0c-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.654286 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.654260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f92d6e24-f969-4434-82fc-a5a7726adf0c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.654418 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.654344    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f92d6e24-f969-4434-82fc-a5a7726adf0c-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.655786 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.655764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f92d6e24-f969-4434-82fc-a5a7726adf0c-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.663275 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.663211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4cnh\" (UniqueName: \"kubernetes.io/projected/f92d6e24-f969-4434-82fc-a5a7726adf0c-kube-api-access-n4cnh\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.721123 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.721093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:20.994684 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:20.994661    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"]
Sep 17 17:27:20.997058 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:27:20.997031    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92d6e24_f969_4434_82fc_a5a7726adf0c.slice/crio-2931ba85465759790a1fa153adc075fa034c6c3787c56fc580bff02f30aeeb54 WatchSource:0}: Error finding container 2931ba85465759790a1fa153adc075fa034c6c3787c56fc580bff02f30aeeb54: Status 404 returned error can't find the container with id 2931ba85465759790a1fa153adc075fa034c6c3787c56fc580bff02f30aeeb54
Sep 17 17:27:21.019096 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:21.019078    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc_239ad695-2889-426d-b199-470808c290b3/storage-initializer/0.log"
Sep 17 17:27:21.019422 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:21.019176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc" event={"ID":"239ad695-2889-426d-b199-470808c290b3","Type":"ContainerStarted","Data":"9b5c7d01f569c8fb4cb6041938db763c398973cf6f81be805b4fb5d4572d8699"}
Sep 17 17:27:21.020368 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:21.020341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" event={"ID":"f92d6e24-f969-4434-82fc-a5a7726adf0c","Type":"ContainerStarted","Data":"2931ba85465759790a1fa153adc075fa034c6c3787c56fc580bff02f30aeeb54"}
Sep 17 17:27:22.026823 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:22.026782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" event={"ID":"f92d6e24-f969-4434-82fc-a5a7726adf0c","Type":"ContainerStarted","Data":"ce00b963d09275cff883db66ab253e5558854c51b83c11f1b965ceaee08fc5bf"}
Sep 17 17:27:24.037252 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:24.037223    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc_239ad695-2889-426d-b199-470808c290b3/storage-initializer/1.log"
Sep 17 17:27:24.037656 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:24.037611    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc_239ad695-2889-426d-b199-470808c290b3/storage-initializer/0.log"
Sep 17 17:27:24.037656 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:24.037644    2577 generic.go:356] "Generic (PLEG): container finished" podID="239ad695-2889-426d-b199-470808c290b3" containerID="9b5c7d01f569c8fb4cb6041938db763c398973cf6f81be805b4fb5d4572d8699" exitCode=1
Sep 17 17:27:24.037732 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:24.037716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc" event={"ID":"239ad695-2889-426d-b199-470808c290b3","Type":"ContainerDied","Data":"9b5c7d01f569c8fb4cb6041938db763c398973cf6f81be805b4fb5d4572d8699"}
Sep 17 17:27:24.037789 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:24.037778    2577 scope.go:117] "RemoveContainer" containerID="4c590c5df80cc0057eaba3252bc3521843f1edbc74a40a265bda699f0bb9858f"
Sep 17 17:27:24.038079 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:24.038062    2577 scope.go:117] "RemoveContainer" containerID="4c590c5df80cc0057eaba3252bc3521843f1edbc74a40a265bda699f0bb9858f"
Sep 17 17:27:24.048397 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:27:24.048373    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc_kserve-ci-e2e-test(239ad695-2889-426d-b199-470808c290b3)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc" podUID="239ad695-2889-426d-b199-470808c290b3"
Sep 17 17:27:24.048553 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:27:24.048524    2577 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc_kserve-ci-e2e-test_239ad695-2889-426d-b199-470808c290b3_0 in pod sandbox 2bb20eab7ceb2f5ab9aa78f0b4e433b0abd6779a82dced3559d7da65dbef8aa8 from index: no such id: '4c590c5df80cc0057eaba3252bc3521843f1edbc74a40a265bda699f0bb9858f'" containerID="4c590c5df80cc0057eaba3252bc3521843f1edbc74a40a265bda699f0bb9858f"
Sep 17 17:27:24.048646 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:24.048562    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c590c5df80cc0057eaba3252bc3521843f1edbc74a40a265bda699f0bb9858f"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc_kserve-ci-e2e-test_239ad695-2889-426d-b199-470808c290b3_0 in pod sandbox 2bb20eab7ceb2f5ab9aa78f0b4e433b0abd6779a82dced3559d7da65dbef8aa8 from index: no such id: '4c590c5df80cc0057eaba3252bc3521843f1edbc74a40a265bda699f0bb9858f'"
Sep 17 17:27:25.044070 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.044044    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc_239ad695-2889-426d-b199-470808c290b3/storage-initializer/1.log"
Sep 17 17:27:25.270688 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.270653    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"]
Sep 17 17:27:25.404348 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.404328    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc_239ad695-2889-426d-b199-470808c290b3/storage-initializer/1.log"
Sep 17 17:27:25.407880 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.407862    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:25.484580 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.484551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/239ad695-2889-426d-b199-470808c290b3-proxy-tls\") pod \"239ad695-2889-426d-b199-470808c290b3\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") "
Sep 17 17:27:25.484580 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.484583    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/239ad695-2889-426d-b199-470808c290b3-kube-rbac-proxy-sar-config\") pod \"239ad695-2889-426d-b199-470808c290b3\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") "
Sep 17 17:27:25.484739 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.484616    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/239ad695-2889-426d-b199-470808c290b3-kserve-provision-location\") pod \"239ad695-2889-426d-b199-470808c290b3\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") "
Sep 17 17:27:25.484739 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.484681    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzbzn\" (UniqueName: \"kubernetes.io/projected/239ad695-2889-426d-b199-470808c290b3-kube-api-access-kzbzn\") pod \"239ad695-2889-426d-b199-470808c290b3\" (UID: \"239ad695-2889-426d-b199-470808c290b3\") "
Sep 17 17:27:25.484946 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.484926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/239ad695-2889-426d-b199-470808c290b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "239ad695-2889-426d-b199-470808c290b3" (UID: "239ad695-2889-426d-b199-470808c290b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:27:25.485069 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.485045    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239ad695-2889-426d-b199-470808c290b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "239ad695-2889-426d-b199-470808c290b3" (UID: "239ad695-2889-426d-b199-470808c290b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:27:25.486627 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.486604    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/239ad695-2889-426d-b199-470808c290b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "239ad695-2889-426d-b199-470808c290b3" (UID: "239ad695-2889-426d-b199-470808c290b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:27:25.486723 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.486685    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239ad695-2889-426d-b199-470808c290b3-kube-api-access-kzbzn" (OuterVolumeSpecName: "kube-api-access-kzbzn") pod "239ad695-2889-426d-b199-470808c290b3" (UID: "239ad695-2889-426d-b199-470808c290b3"). InnerVolumeSpecName "kube-api-access-kzbzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:27:25.585232 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.585175    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzbzn\" (UniqueName: \"kubernetes.io/projected/239ad695-2889-426d-b199-470808c290b3-kube-api-access-kzbzn\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:27:25.585232 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.585196    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/239ad695-2889-426d-b199-470808c290b3-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:27:25.585232 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.585205    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/239ad695-2889-426d-b199-470808c290b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:27:25.585232 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:25.585214    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/239ad695-2889-426d-b199-470808c290b3-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:27:26.049761 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.049738    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc_239ad695-2889-426d-b199-470808c290b3/storage-initializer/1.log"
Sep 17 17:27:26.050232 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.049819    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc" event={"ID":"239ad695-2889-426d-b199-470808c290b3","Type":"ContainerDied","Data":"2bb20eab7ceb2f5ab9aa78f0b4e433b0abd6779a82dced3559d7da65dbef8aa8"}
Sep 17 17:27:26.050232 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.049837    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"
Sep 17 17:27:26.050232 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.049869    2577 scope.go:117] "RemoveContainer" containerID="9b5c7d01f569c8fb4cb6041938db763c398973cf6f81be805b4fb5d4572d8699"
Sep 17 17:27:26.051702 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.051680    2577 generic.go:356] "Generic (PLEG): container finished" podID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerID="ce00b963d09275cff883db66ab253e5558854c51b83c11f1b965ceaee08fc5bf" exitCode=0
Sep 17 17:27:26.051820 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.051769    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" event={"ID":"f92d6e24-f969-4434-82fc-a5a7726adf0c","Type":"ContainerDied","Data":"ce00b963d09275cff883db66ab253e5558854c51b83c11f1b965ceaee08fc5bf"}
Sep 17 17:27:26.100944 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.100919    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"]
Sep 17 17:27:26.107279 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.107256    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78b8757485-5dfdc"]
Sep 17 17:27:26.426128 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.426061    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"]
Sep 17 17:27:26.426395 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.426383    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="239ad695-2889-426d-b199-470808c290b3" containerName="storage-initializer"
Sep 17 17:27:26.426441 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.426398    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="239ad695-2889-426d-b199-470808c290b3" containerName="storage-initializer"
Sep 17 17:27:26.426441 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.426411    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="239ad695-2889-426d-b199-470808c290b3" containerName="storage-initializer"
Sep 17 17:27:26.426441 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.426416    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="239ad695-2889-426d-b199-470808c290b3" containerName="storage-initializer"
Sep 17 17:27:26.426549 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.426478    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="239ad695-2889-426d-b199-470808c290b3" containerName="storage-initializer"
Sep 17 17:27:26.426549 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.426489    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="239ad695-2889-426d-b199-470808c290b3" containerName="storage-initializer"
Sep 17 17:27:26.429524 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.429508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.432448 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.432426    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 17 17:27:26.432448 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.432436    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 17 17:27:26.432623 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.432505    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 17:27:26.449018 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.448983    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="239ad695-2889-426d-b199-470808c290b3" path="/var/lib/kubelet/pods/239ad695-2889-426d-b199-470808c290b3/volumes"
Sep 17 17:27:26.449312 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.449293    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"]
Sep 17 17:27:26.490390 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.490366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.490518 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.490415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.490518 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.490445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.490518 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.490487    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.490675 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.490540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqkq4\" (UniqueName: \"kubernetes.io/projected/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kube-api-access-rqkq4\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.591341 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.591315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.591441 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.591357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.591441 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.591395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.591441 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.591427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.591649 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.591627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqkq4\" (UniqueName: \"kubernetes.io/projected/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kube-api-access-rqkq4\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.591860 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.591836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.592044 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.592026    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.592097 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.592044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.593890 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.593874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.599696 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.599673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqkq4\" (UniqueName: \"kubernetes.io/projected/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kube-api-access-rqkq4\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:26.739615 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:26.739555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:27.013130 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:27.013105    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"]
Sep 17 17:27:27.014965 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:27:27.014936    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d26a320_4308_4fd4_9e09_d7dcbdb8d2b9.slice/crio-2cf35002a84dce3f5803697bf2ab3be71dc80103f6a3c147a9b7728db8cc9a45 WatchSource:0}: Error finding container 2cf35002a84dce3f5803697bf2ab3be71dc80103f6a3c147a9b7728db8cc9a45: Status 404 returned error can't find the container with id 2cf35002a84dce3f5803697bf2ab3be71dc80103f6a3c147a9b7728db8cc9a45
Sep 17 17:27:27.060206 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:27.060182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" event={"ID":"f92d6e24-f969-4434-82fc-a5a7726adf0c","Type":"ContainerStarted","Data":"bd9ec0ca597509c79b70afde958af80dc5bf56a85fae874da8982ed59a676c4e"}
Sep 17 17:27:27.060587 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:27.060242    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" event={"ID":"f92d6e24-f969-4434-82fc-a5a7726adf0c","Type":"ContainerStarted","Data":"400e22f8dbc9855372414d497920b7aef170d0ce31e394ada9ea6376fbdf867a"}
Sep 17 17:27:27.060587 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:27.060450    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:27.061388 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:27.061366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" event={"ID":"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9","Type":"ContainerStarted","Data":"2cf35002a84dce3f5803697bf2ab3be71dc80103f6a3c147a9b7728db8cc9a45"}
Sep 17 17:27:27.080730 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:27.080692    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" podStartSLOduration=2.368573998 podStartE2EDuration="7.080679037s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.712105039s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:27:27.080403429 +0000 UTC m=+2765.105311744" watchObservedRunningTime="2026-09-17 17:27:27.080679037 +0000 UTC m=+2765.105587324"
Sep 17 17:27:28.068109 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:28.068064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" event={"ID":"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9","Type":"ContainerStarted","Data":"b2bda870dc9575940beb6fd6432e7fae33dc236db7744a8c10e5985f06087d9f"}
Sep 17 17:27:28.068585 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:28.068469    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:28.069563 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:28.069538    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 17:27:29.075388 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:29.075353    2577 generic.go:356] "Generic (PLEG): container finished" podID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerID="b2bda870dc9575940beb6fd6432e7fae33dc236db7744a8c10e5985f06087d9f" exitCode=0
Sep 17 17:27:29.075816 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:29.075430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" event={"ID":"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9","Type":"ContainerDied","Data":"b2bda870dc9575940beb6fd6432e7fae33dc236db7744a8c10e5985f06087d9f"}
Sep 17 17:27:29.075919 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:29.075888    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 17:27:30.083124 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:30.083084    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" event={"ID":"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9","Type":"ContainerStarted","Data":"5c39e9dd0108c16e9a244cba622edadf177394c5b540fc614a5e713564f29894"}
Sep 17 17:27:30.083556 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:30.083148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" event={"ID":"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9","Type":"ContainerStarted","Data":"a3cd5c8b94ff5abb058aee3cb792f958aa274f4f32730b2d715d79c9e9c98e4c"}
Sep 17 17:27:30.083556 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:30.083266    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:30.102066 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:30.102025    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" podStartSLOduration=3.024029531 podStartE2EDuration="4.10201418s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.077984649s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:27:30.100343618 +0000 UTC m=+2768.125251907" watchObservedRunningTime="2026-09-17 17:27:30.10201418 +0000 UTC m=+2768.126922468"
Sep 17 17:27:31.093105 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:31.093059    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:31.094650 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:31.094624    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 17:27:32.096953 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:32.096909    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 17:27:34.081260 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:34.081231    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:27:34.081730 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:34.081704    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 17:27:37.101594 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:37.101561    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:27:37.102063 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:37.102034    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 17:27:44.082238 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:44.082191    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 17:27:47.102791 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:47.102703    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 17:27:54.082291 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:54.082247    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 17:27:57.102723 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:27:57.102680    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 17:28:04.081698 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:04.081655    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 17:28:07.102887 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:07.102847    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 17:28:14.083294 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:14.083265    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:28:17.102695 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:17.102656    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 17:28:22.043425 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:22.043388    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"]
Sep 17 17:28:22.043828 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:22.043626    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kserve-container" containerID="cri-o://400e22f8dbc9855372414d497920b7aef170d0ce31e394ada9ea6376fbdf867a" gracePeriod=30
Sep 17 17:28:22.043828 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:22.043662    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kube-rbac-proxy" containerID="cri-o://bd9ec0ca597509c79b70afde958af80dc5bf56a85fae874da8982ed59a676c4e" gracePeriod=30
Sep 17 17:28:22.328863 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:22.328786    2577 generic.go:356] "Generic (PLEG): container finished" podID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerID="bd9ec0ca597509c79b70afde958af80dc5bf56a85fae874da8982ed59a676c4e" exitCode=2
Sep 17 17:28:22.328997 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:22.328850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" event={"ID":"f92d6e24-f969-4434-82fc-a5a7726adf0c","Type":"ContainerDied","Data":"bd9ec0ca597509c79b70afde958af80dc5bf56a85fae874da8982ed59a676c4e"}
Sep 17 17:28:24.076988 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:24.076936    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 17 17:28:24.082400 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:24.082360    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 17:28:24.887690 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:24.887661    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:28:25.050140 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.050113    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f92d6e24-f969-4434-82fc-a5a7726adf0c-kserve-provision-location\") pod \"f92d6e24-f969-4434-82fc-a5a7726adf0c\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") "
Sep 17 17:28:25.050413 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.050152    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f92d6e24-f969-4434-82fc-a5a7726adf0c-proxy-tls\") pod \"f92d6e24-f969-4434-82fc-a5a7726adf0c\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") "
Sep 17 17:28:25.050413 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.050209    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f92d6e24-f969-4434-82fc-a5a7726adf0c-kube-rbac-proxy-sar-config\") pod \"f92d6e24-f969-4434-82fc-a5a7726adf0c\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") "
Sep 17 17:28:25.050413 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.050236    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f92d6e24-f969-4434-82fc-a5a7726adf0c-cabundle-cert\") pod \"f92d6e24-f969-4434-82fc-a5a7726adf0c\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") "
Sep 17 17:28:25.050413 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.050265    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4cnh\" (UniqueName: \"kubernetes.io/projected/f92d6e24-f969-4434-82fc-a5a7726adf0c-kube-api-access-n4cnh\") pod \"f92d6e24-f969-4434-82fc-a5a7726adf0c\" (UID: \"f92d6e24-f969-4434-82fc-a5a7726adf0c\") "
Sep 17 17:28:25.050692 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.050478    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f92d6e24-f969-4434-82fc-a5a7726adf0c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f92d6e24-f969-4434-82fc-a5a7726adf0c" (UID: "f92d6e24-f969-4434-82fc-a5a7726adf0c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:28:25.050692 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.050569    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f92d6e24-f969-4434-82fc-a5a7726adf0c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f92d6e24-f969-4434-82fc-a5a7726adf0c" (UID: "f92d6e24-f969-4434-82fc-a5a7726adf0c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:28:25.052225 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.052154    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f92d6e24-f969-4434-82fc-a5a7726adf0c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f92d6e24-f969-4434-82fc-a5a7726adf0c" (UID: "f92d6e24-f969-4434-82fc-a5a7726adf0c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:28:25.052328 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.052266    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92d6e24-f969-4434-82fc-a5a7726adf0c-kube-api-access-n4cnh" (OuterVolumeSpecName: "kube-api-access-n4cnh") pod "f92d6e24-f969-4434-82fc-a5a7726adf0c" (UID: "f92d6e24-f969-4434-82fc-a5a7726adf0c"). InnerVolumeSpecName "kube-api-access-n4cnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:28:25.056623 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.056602    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92d6e24-f969-4434-82fc-a5a7726adf0c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f92d6e24-f969-4434-82fc-a5a7726adf0c" (UID: "f92d6e24-f969-4434-82fc-a5a7726adf0c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:28:25.150738 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.150715    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f92d6e24-f969-4434-82fc-a5a7726adf0c-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:28:25.150738 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.150736    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f92d6e24-f969-4434-82fc-a5a7726adf0c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:28:25.151115 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.150745    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f92d6e24-f969-4434-82fc-a5a7726adf0c-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:28:25.151115 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.150754    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4cnh\" (UniqueName: \"kubernetes.io/projected/f92d6e24-f969-4434-82fc-a5a7726adf0c-kube-api-access-n4cnh\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:28:25.151115 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.150762    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f92d6e24-f969-4434-82fc-a5a7726adf0c-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:28:25.343193 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.343102    2577 generic.go:356] "Generic (PLEG): container finished" podID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerID="400e22f8dbc9855372414d497920b7aef170d0ce31e394ada9ea6376fbdf867a" exitCode=0
Sep 17 17:28:25.343193 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.343185    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"
Sep 17 17:28:25.343346 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.343191    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" event={"ID":"f92d6e24-f969-4434-82fc-a5a7726adf0c","Type":"ContainerDied","Data":"400e22f8dbc9855372414d497920b7aef170d0ce31e394ada9ea6376fbdf867a"}
Sep 17 17:28:25.343346 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.343288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r" event={"ID":"f92d6e24-f969-4434-82fc-a5a7726adf0c","Type":"ContainerDied","Data":"2931ba85465759790a1fa153adc075fa034c6c3787c56fc580bff02f30aeeb54"}
Sep 17 17:28:25.343346 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.343316    2577 scope.go:117] "RemoveContainer" containerID="bd9ec0ca597509c79b70afde958af80dc5bf56a85fae874da8982ed59a676c4e"
Sep 17 17:28:25.352313 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.352292    2577 scope.go:117] "RemoveContainer" containerID="400e22f8dbc9855372414d497920b7aef170d0ce31e394ada9ea6376fbdf867a"
Sep 17 17:28:25.359418 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.359400    2577 scope.go:117] "RemoveContainer" containerID="ce00b963d09275cff883db66ab253e5558854c51b83c11f1b965ceaee08fc5bf"
Sep 17 17:28:25.364526 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.364505    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"]
Sep 17 17:28:25.366482 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.366457    2577 scope.go:117] "RemoveContainer" containerID="bd9ec0ca597509c79b70afde958af80dc5bf56a85fae874da8982ed59a676c4e"
Sep 17 17:28:25.366726 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:28:25.366707    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9ec0ca597509c79b70afde958af80dc5bf56a85fae874da8982ed59a676c4e\": container with ID starting with bd9ec0ca597509c79b70afde958af80dc5bf56a85fae874da8982ed59a676c4e not found: ID does not exist" containerID="bd9ec0ca597509c79b70afde958af80dc5bf56a85fae874da8982ed59a676c4e"
Sep 17 17:28:25.366832 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.366738    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9ec0ca597509c79b70afde958af80dc5bf56a85fae874da8982ed59a676c4e"} err="failed to get container status \"bd9ec0ca597509c79b70afde958af80dc5bf56a85fae874da8982ed59a676c4e\": rpc error: code = NotFound desc = could not find container \"bd9ec0ca597509c79b70afde958af80dc5bf56a85fae874da8982ed59a676c4e\": container with ID starting with bd9ec0ca597509c79b70afde958af80dc5bf56a85fae874da8982ed59a676c4e not found: ID does not exist"
Sep 17 17:28:25.366832 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.366763    2577 scope.go:117] "RemoveContainer" containerID="400e22f8dbc9855372414d497920b7aef170d0ce31e394ada9ea6376fbdf867a"
Sep 17 17:28:25.367311 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:28:25.367021    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400e22f8dbc9855372414d497920b7aef170d0ce31e394ada9ea6376fbdf867a\": container with ID starting with 400e22f8dbc9855372414d497920b7aef170d0ce31e394ada9ea6376fbdf867a not found: ID does not exist" containerID="400e22f8dbc9855372414d497920b7aef170d0ce31e394ada9ea6376fbdf867a"
Sep 17 17:28:25.367311 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.367043    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400e22f8dbc9855372414d497920b7aef170d0ce31e394ada9ea6376fbdf867a"} err="failed to get container status \"400e22f8dbc9855372414d497920b7aef170d0ce31e394ada9ea6376fbdf867a\": rpc error: code = NotFound desc = could not find container \"400e22f8dbc9855372414d497920b7aef170d0ce31e394ada9ea6376fbdf867a\": container with ID starting with 400e22f8dbc9855372414d497920b7aef170d0ce31e394ada9ea6376fbdf867a not found: ID does not exist"
Sep 17 17:28:25.367311 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.367060    2577 scope.go:117] "RemoveContainer" containerID="ce00b963d09275cff883db66ab253e5558854c51b83c11f1b965ceaee08fc5bf"
Sep 17 17:28:25.367461 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:28:25.367442    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce00b963d09275cff883db66ab253e5558854c51b83c11f1b965ceaee08fc5bf\": container with ID starting with ce00b963d09275cff883db66ab253e5558854c51b83c11f1b965ceaee08fc5bf not found: ID does not exist" containerID="ce00b963d09275cff883db66ab253e5558854c51b83c11f1b965ceaee08fc5bf"
Sep 17 17:28:25.367506 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.367461    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce00b963d09275cff883db66ab253e5558854c51b83c11f1b965ceaee08fc5bf"} err="failed to get container status \"ce00b963d09275cff883db66ab253e5558854c51b83c11f1b965ceaee08fc5bf\": rpc error: code = NotFound desc = could not find container \"ce00b963d09275cff883db66ab253e5558854c51b83c11f1b965ceaee08fc5bf\": container with ID starting with ce00b963d09275cff883db66ab253e5558854c51b83c11f1b965ceaee08fc5bf not found: ID does not exist"
Sep 17 17:28:25.368671 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:25.368653    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jhs5r"]
Sep 17 17:28:26.446785 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:26.446741    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" path="/var/lib/kubelet/pods/f92d6e24-f969-4434-82fc-a5a7726adf0c/volumes"
Sep 17 17:28:27.102793 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.102754    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 17:28:27.217956 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.217924    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"]
Sep 17 17:28:27.218259 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.218247    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="storage-initializer"
Sep 17 17:28:27.218315 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.218261    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="storage-initializer"
Sep 17 17:28:27.218315 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.218273    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kserve-container"
Sep 17 17:28:27.218315 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.218279    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kserve-container"
Sep 17 17:28:27.218315 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.218284    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kube-rbac-proxy"
Sep 17 17:28:27.218315 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.218289    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kube-rbac-proxy"
Sep 17 17:28:27.218500 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.218339    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kserve-container"
Sep 17 17:28:27.218500 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.218350    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f92d6e24-f969-4434-82fc-a5a7726adf0c" containerName="kube-rbac-proxy"
Sep 17 17:28:27.222859 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.222840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.225368 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.225343    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 17 17:28:27.225503 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.225487    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:28:27.225599 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.225585    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 17 17:28:27.231620 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.231598    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"]
Sep 17 17:28:27.370375 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.370309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd37b957-140b-46ff-96aa-f0d57523183f-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-fttxq\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.370375 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.370357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkxx7\" (UniqueName: \"kubernetes.io/projected/cd37b957-140b-46ff-96aa-f0d57523183f-kube-api-access-qkxx7\") pod \"isvc-pmml-predictor-8674795f59-fttxq\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.370551 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.370390    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd37b957-140b-46ff-96aa-f0d57523183f-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-fttxq\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.370551 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.370434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd37b957-140b-46ff-96aa-f0d57523183f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-fttxq\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.370551 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.370466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd37b957-140b-46ff-96aa-f0d57523183f-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-fttxq\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.471280 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.471250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd37b957-140b-46ff-96aa-f0d57523183f-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-fttxq\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.471647 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.471283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd37b957-140b-46ff-96aa-f0d57523183f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-fttxq\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.471647 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.471316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd37b957-140b-46ff-96aa-f0d57523183f-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-fttxq\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.471647 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.471343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd37b957-140b-46ff-96aa-f0d57523183f-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-fttxq\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.471647 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.471382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkxx7\" (UniqueName: \"kubernetes.io/projected/cd37b957-140b-46ff-96aa-f0d57523183f-kube-api-access-qkxx7\") pod \"isvc-pmml-predictor-8674795f59-fttxq\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.471647 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.471613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd37b957-140b-46ff-96aa-f0d57523183f-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-fttxq\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.472007 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.471988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd37b957-140b-46ff-96aa-f0d57523183f-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-fttxq\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.472100 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.472083    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd37b957-140b-46ff-96aa-f0d57523183f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-fttxq\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.473826 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.473809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd37b957-140b-46ff-96aa-f0d57523183f-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-fttxq\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.479228 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.479197    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkxx7\" (UniqueName: \"kubernetes.io/projected/cd37b957-140b-46ff-96aa-f0d57523183f-kube-api-access-qkxx7\") pod \"isvc-pmml-predictor-8674795f59-fttxq\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.533147 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.533123    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:27.804114 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:27.804048    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"]
Sep 17 17:28:27.806264 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:28:27.806238    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd37b957_140b_46ff_96aa_f0d57523183f.slice/crio-61d91d1dc1c0b9103e6d93d28f0f200984012cd046bf8c17970434d6b8efe835 WatchSource:0}: Error finding container 61d91d1dc1c0b9103e6d93d28f0f200984012cd046bf8c17970434d6b8efe835: Status 404 returned error can't find the container with id 61d91d1dc1c0b9103e6d93d28f0f200984012cd046bf8c17970434d6b8efe835
Sep 17 17:28:28.358554 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:28.358519    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" event={"ID":"cd37b957-140b-46ff-96aa-f0d57523183f","Type":"ContainerStarted","Data":"8fb3e468926c99bd89e86b798609832bf34395a731231c98bb8945a4efe6fd58"}
Sep 17 17:28:28.358758 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:28.358608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" event={"ID":"cd37b957-140b-46ff-96aa-f0d57523183f","Type":"ContainerStarted","Data":"61d91d1dc1c0b9103e6d93d28f0f200984012cd046bf8c17970434d6b8efe835"}
Sep 17 17:28:32.378181 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:32.378128    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd37b957-140b-46ff-96aa-f0d57523183f" containerID="8fb3e468926c99bd89e86b798609832bf34395a731231c98bb8945a4efe6fd58" exitCode=0
Sep 17 17:28:32.378586 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:32.378204    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" event={"ID":"cd37b957-140b-46ff-96aa-f0d57523183f","Type":"ContainerDied","Data":"8fb3e468926c99bd89e86b798609832bf34395a731231c98bb8945a4efe6fd58"}
Sep 17 17:28:32.379522 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:32.379503    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:28:37.102240 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:37.102203    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 17:28:40.430403 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:40.430370    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" event={"ID":"cd37b957-140b-46ff-96aa-f0d57523183f","Type":"ContainerStarted","Data":"fe803f8987deb7ec0d95c0bc45f8218f00524298e34d88ece7c32752239b2344"}
Sep 17 17:28:40.430798 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:40.430453    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" event={"ID":"cd37b957-140b-46ff-96aa-f0d57523183f","Type":"ContainerStarted","Data":"96aef3e004ff45ac29587318d2b321121600bd3b9d09cba6f4cae798c0137db3"}
Sep 17 17:28:40.430798 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:40.430734    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:40.430896 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:40.430880    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:40.432240 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:40.432208    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 17:28:40.450170 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:40.450126    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" podStartSLOduration=2.502185692 podStartE2EDuration="13.45011441s" totalImagesPullingTime="7.256009423s" totalInitContainerRuntime="3.691919295s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:28:40.447865753 +0000 UTC m=+2838.472774041" watchObservedRunningTime="2026-09-17 17:28:40.45011441 +0000 UTC m=+2838.475022697"
Sep 17 17:28:41.435007 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:41.434961    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 17:28:42.439720 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:42.439680    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 17:28:47.102313 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:47.102284    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:28:47.444012 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:47.443938    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:28:47.444444 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:47.444416    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 17:28:56.432082 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:56.432051    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"]
Sep 17 17:28:56.432657 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:56.432337    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kserve-container" containerID="cri-o://a3cd5c8b94ff5abb058aee3cb792f958aa274f4f32730b2d715d79c9e9c98e4c" gracePeriod=30
Sep 17 17:28:56.432657 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:56.432380    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kube-rbac-proxy" containerID="cri-o://5c39e9dd0108c16e9a244cba622edadf177394c5b540fc614a5e713564f29894" gracePeriod=30
Sep 17 17:28:57.097824 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:57.097783    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Sep 17 17:28:57.102756 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:57.102727    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 17:28:57.444962 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:57.444873    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 17:28:57.505794 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:57.505752    2577 generic.go:356] "Generic (PLEG): container finished" podID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerID="5c39e9dd0108c16e9a244cba622edadf177394c5b540fc614a5e713564f29894" exitCode=2
Sep 17 17:28:57.505971 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:28:57.505807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" event={"ID":"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9","Type":"ContainerDied","Data":"5c39e9dd0108c16e9a244cba622edadf177394c5b540fc614a5e713564f29894"}
Sep 17 17:29:00.678498 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.678475    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:29:00.729037 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.729015    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqkq4\" (UniqueName: \"kubernetes.io/projected/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kube-api-access-rqkq4\") pod \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") "
Sep 17 17:29:00.729207 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.729063    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-cabundle-cert\") pod \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") "
Sep 17 17:29:00.729207 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.729092    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kserve-provision-location\") pod \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") "
Sep 17 17:29:00.729207 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.729107    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kube-rbac-proxy-sar-config\") pod \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") "
Sep 17 17:29:00.729207 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.729129    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-proxy-tls\") pod \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\" (UID: \"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9\") "
Sep 17 17:29:00.729453 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.729399    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" (UID: "3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:29:00.729507 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.729463    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" (UID: "3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:29:00.729507 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.729475    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" (UID: "3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:29:00.731091 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.731066    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kube-api-access-rqkq4" (OuterVolumeSpecName: "kube-api-access-rqkq4") pod "3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" (UID: "3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9"). InnerVolumeSpecName "kube-api-access-rqkq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:29:00.731210 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.731093    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" (UID: "3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:29:00.829893 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.829869    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqkq4\" (UniqueName: \"kubernetes.io/projected/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kube-api-access-rqkq4\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:29:00.830018 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.829895    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:29:00.830018 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.829909    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:29:00.830018 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.829921    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:29:00.830018 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:00.829933    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:29:01.525271 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:01.525229    2577 generic.go:356] "Generic (PLEG): container finished" podID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerID="a3cd5c8b94ff5abb058aee3cb792f958aa274f4f32730b2d715d79c9e9c98e4c" exitCode=0
Sep 17 17:29:01.525445 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:01.525290    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"
Sep 17 17:29:01.525445 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:01.525304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" event={"ID":"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9","Type":"ContainerDied","Data":"a3cd5c8b94ff5abb058aee3cb792f958aa274f4f32730b2d715d79c9e9c98e4c"}
Sep 17 17:29:01.525445 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:01.525358    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw" event={"ID":"3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9","Type":"ContainerDied","Data":"2cf35002a84dce3f5803697bf2ab3be71dc80103f6a3c147a9b7728db8cc9a45"}
Sep 17 17:29:01.525445 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:01.525386    2577 scope.go:117] "RemoveContainer" containerID="5c39e9dd0108c16e9a244cba622edadf177394c5b540fc614a5e713564f29894"
Sep 17 17:29:01.534302 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:01.534287    2577 scope.go:117] "RemoveContainer" containerID="a3cd5c8b94ff5abb058aee3cb792f958aa274f4f32730b2d715d79c9e9c98e4c"
Sep 17 17:29:01.541262 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:01.541246    2577 scope.go:117] "RemoveContainer" containerID="b2bda870dc9575940beb6fd6432e7fae33dc236db7744a8c10e5985f06087d9f"
Sep 17 17:29:01.547030 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:01.547007    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"]
Sep 17 17:29:01.548817 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:01.548793    2577 scope.go:117] "RemoveContainer" containerID="5c39e9dd0108c16e9a244cba622edadf177394c5b540fc614a5e713564f29894"
Sep 17 17:29:01.549070 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:29:01.549052    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c39e9dd0108c16e9a244cba622edadf177394c5b540fc614a5e713564f29894\": container with ID starting with 5c39e9dd0108c16e9a244cba622edadf177394c5b540fc614a5e713564f29894 not found: ID does not exist" containerID="5c39e9dd0108c16e9a244cba622edadf177394c5b540fc614a5e713564f29894"
Sep 17 17:29:01.549120 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:01.549079    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c39e9dd0108c16e9a244cba622edadf177394c5b540fc614a5e713564f29894"} err="failed to get container status \"5c39e9dd0108c16e9a244cba622edadf177394c5b540fc614a5e713564f29894\": rpc error: code = NotFound desc = could not find container \"5c39e9dd0108c16e9a244cba622edadf177394c5b540fc614a5e713564f29894\": container with ID starting with 5c39e9dd0108c16e9a244cba622edadf177394c5b540fc614a5e713564f29894 not found: ID does not exist"
Sep 17 17:29:01.549120 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:01.549096    2577 scope.go:117] "RemoveContainer" containerID="a3cd5c8b94ff5abb058aee3cb792f958aa274f4f32730b2d715d79c9e9c98e4c"
Sep 17 17:29:01.549372 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:29:01.549350    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3cd5c8b94ff5abb058aee3cb792f958aa274f4f32730b2d715d79c9e9c98e4c\": container with ID starting with a3cd5c8b94ff5abb058aee3cb792f958aa274f4f32730b2d715d79c9e9c98e4c not found: ID does not exist" containerID="a3cd5c8b94ff5abb058aee3cb792f958aa274f4f32730b2d715d79c9e9c98e4c"
Sep 17 17:29:01.549419 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:01.549380    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cd5c8b94ff5abb058aee3cb792f958aa274f4f32730b2d715d79c9e9c98e4c"} err="failed to get container status \"a3cd5c8b94ff5abb058aee3cb792f958aa274f4f32730b2d715d79c9e9c98e4c\": rpc error: code = NotFound desc = could not find container \"a3cd5c8b94ff5abb058aee3cb792f958aa274f4f32730b2d715d79c9e9c98e4c\": container with ID starting with a3cd5c8b94ff5abb058aee3cb792f958aa274f4f32730b2d715d79c9e9c98e4c not found: ID does not exist"
Sep 17 17:29:01.549419 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:01.549402    2577 scope.go:117] "RemoveContainer" containerID="b2bda870dc9575940beb6fd6432e7fae33dc236db7744a8c10e5985f06087d9f"
Sep 17 17:29:01.549653 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:29:01.549636    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2bda870dc9575940beb6fd6432e7fae33dc236db7744a8c10e5985f06087d9f\": container with ID starting with b2bda870dc9575940beb6fd6432e7fae33dc236db7744a8c10e5985f06087d9f not found: ID does not exist" containerID="b2bda870dc9575940beb6fd6432e7fae33dc236db7744a8c10e5985f06087d9f"
Sep 17 17:29:01.549695 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:01.549660    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2bda870dc9575940beb6fd6432e7fae33dc236db7744a8c10e5985f06087d9f"} err="failed to get container status \"b2bda870dc9575940beb6fd6432e7fae33dc236db7744a8c10e5985f06087d9f\": rpc error: code = NotFound desc = could not find container \"b2bda870dc9575940beb6fd6432e7fae33dc236db7744a8c10e5985f06087d9f\": container with ID starting with b2bda870dc9575940beb6fd6432e7fae33dc236db7744a8c10e5985f06087d9f not found: ID does not exist"
Sep 17 17:29:01.550200 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:01.550184    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5c4d5f94bf-r9nnw"]
Sep 17 17:29:02.446583 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:02.446554    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" path="/var/lib/kubelet/pods/3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9/volumes"
Sep 17 17:29:07.445185 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:07.445133    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 17:29:08.633724 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.633684    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"]
Sep 17 17:29:08.634199 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.634133    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kserve-container"
Sep 17 17:29:08.634199 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.634151    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kserve-container"
Sep 17 17:29:08.634199 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.634194    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="storage-initializer"
Sep 17 17:29:08.634392 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.634205    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="storage-initializer"
Sep 17 17:29:08.634392 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.634229    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kube-rbac-proxy"
Sep 17 17:29:08.634392 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.634238    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kube-rbac-proxy"
Sep 17 17:29:08.634392 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.634328    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kube-rbac-proxy"
Sep 17 17:29:08.634392 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.634340    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d26a320-4308-4fd4-9e09-d7dcbdb8d2b9" containerName="kserve-container"
Sep 17 17:29:08.639842 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.639821    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:08.642665 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.642641    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 17:29:08.642665 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.642654    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 17 17:29:08.642883 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.642869    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 17 17:29:08.648435 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.648414    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"]
Sep 17 17:29:08.687602 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.687581    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:08.687716 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.687609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:08.687716 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.687653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:08.687803 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.687712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s686n\" (UniqueName: \"kubernetes.io/projected/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kube-api-access-s686n\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:08.687803 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.687740    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:08.788943 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.788901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:08.789097 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.788957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:08.789097 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.789012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:08.789097 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.789031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s686n\" (UniqueName: \"kubernetes.io/projected/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kube-api-access-s686n\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:08.789302 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:29:08.789133    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 17 17:29:08.789302 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.789143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:08.789302 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:29:08.789219    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-proxy-tls podName:eb850cdc-5a9a-4d34-8a92-d47c7a9b576b nodeName:}" failed. No retries permitted until 2026-09-17 17:29:09.289199223 +0000 UTC m=+2867.314107544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" (UID: "eb850cdc-5a9a-4d34-8a92-d47c7a9b576b") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 17 17:29:08.789484 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.789432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:08.789707 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.789685    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:08.789812 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.789748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:08.799572 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:08.799548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s686n\" (UniqueName: \"kubernetes.io/projected/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kube-api-access-s686n\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:09.292290 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:09.292255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:09.294615 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:09.294593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:09.550007 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:09.549940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:09.886553 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:09.886478    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"]
Sep 17 17:29:09.917789 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:29:09.917743    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb850cdc_5a9a_4d34_8a92_d47c7a9b576b.slice/crio-1c56ad99ce999c2e5ea62bf61cc6567626c2890427508344077fce568f5e2dcb WatchSource:0}: Error finding container 1c56ad99ce999c2e5ea62bf61cc6567626c2890427508344077fce568f5e2dcb: Status 404 returned error can't find the container with id 1c56ad99ce999c2e5ea62bf61cc6567626c2890427508344077fce568f5e2dcb
Sep 17 17:29:10.566476 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:10.566429    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" event={"ID":"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b","Type":"ContainerStarted","Data":"c5a1d9f7aa158addcfb30503f3724f9c392adc251852475ab67937f62f67707c"}
Sep 17 17:29:10.566674 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:10.566494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" event={"ID":"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b","Type":"ContainerStarted","Data":"1c56ad99ce999c2e5ea62bf61cc6567626c2890427508344077fce568f5e2dcb"}
Sep 17 17:29:11.573657 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:11.573624    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerID="c5a1d9f7aa158addcfb30503f3724f9c392adc251852475ab67937f62f67707c" exitCode=0
Sep 17 17:29:11.574067 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:11.573713    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" event={"ID":"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b","Type":"ContainerDied","Data":"c5a1d9f7aa158addcfb30503f3724f9c392adc251852475ab67937f62f67707c"}
Sep 17 17:29:12.581523 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:12.581488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" event={"ID":"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b","Type":"ContainerStarted","Data":"38d55c1880493b237377f38ca6d94e359ac12f60b65a2c41a39a4b48bff76628"}
Sep 17 17:29:12.581900 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:12.581562    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:12.581900 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:12.581589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" event={"ID":"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b","Type":"ContainerStarted","Data":"032907a87f3e3c554a998f06047e88ee877dd89ad421de4bf8b2a9f920846a8a"}
Sep 17 17:29:12.581900 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:12.581610    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:12.583108 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:12.583079    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 17:29:12.600759 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:12.600708    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" podStartSLOduration=3.4569842570000002 podStartE2EDuration="4.600691853s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.143707596s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:29:12.599805443 +0000 UTC m=+2870.624713730" watchObservedRunningTime="2026-09-17 17:29:12.600691853 +0000 UTC m=+2870.625600141"
Sep 17 17:29:13.586095 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:13.586048    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 17:29:17.444826 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:17.444736    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 17:29:18.590197 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:18.590168    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:29:18.590674 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:18.590634    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 17:29:27.444786 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:27.444747    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 17:29:28.590655 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:28.590616    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 17:29:37.445212 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:37.445148    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 17:29:38.591356 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:38.591318    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 17:29:47.444783 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:47.444731    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 17:29:48.590644 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:48.590598    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 17:29:53.441820 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:53.441776    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 17:29:58.591269 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:29:58.591228    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 17:30:03.442352 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:03.442313    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:30:08.591150 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:08.591112    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 17:30:08.728269 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:08.728236    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"]
Sep 17 17:30:08.728513 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:08.728481    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kserve-container" containerID="cri-o://96aef3e004ff45ac29587318d2b321121600bd3b9d09cba6f4cae798c0137db3" gracePeriod=30
Sep 17 17:30:08.728617 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:08.728512    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kube-rbac-proxy" containerID="cri-o://fe803f8987deb7ec0d95c0bc45f8218f00524298e34d88ece7c32752239b2344" gracePeriod=30
Sep 17 17:30:09.841294 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:09.841261    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd37b957-140b-46ff-96aa-f0d57523183f" containerID="fe803f8987deb7ec0d95c0bc45f8218f00524298e34d88ece7c32752239b2344" exitCode=2
Sep 17 17:30:09.841689 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:09.841335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" event={"ID":"cd37b957-140b-46ff-96aa-f0d57523183f","Type":"ContainerDied","Data":"fe803f8987deb7ec0d95c0bc45f8218f00524298e34d88ece7c32752239b2344"}
Sep 17 17:30:12.375075 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.375055    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:30:12.462512 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.462458    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd37b957-140b-46ff-96aa-f0d57523183f-cabundle-cert\") pod \"cd37b957-140b-46ff-96aa-f0d57523183f\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") "
Sep 17 17:30:12.462512 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.462502    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd37b957-140b-46ff-96aa-f0d57523183f-kserve-provision-location\") pod \"cd37b957-140b-46ff-96aa-f0d57523183f\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") "
Sep 17 17:30:12.462669 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.462532    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd37b957-140b-46ff-96aa-f0d57523183f-kube-rbac-proxy-sar-config\") pod \"cd37b957-140b-46ff-96aa-f0d57523183f\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") "
Sep 17 17:30:12.462669 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.462553    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkxx7\" (UniqueName: \"kubernetes.io/projected/cd37b957-140b-46ff-96aa-f0d57523183f-kube-api-access-qkxx7\") pod \"cd37b957-140b-46ff-96aa-f0d57523183f\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") "
Sep 17 17:30:12.462669 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.462579    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd37b957-140b-46ff-96aa-f0d57523183f-proxy-tls\") pod \"cd37b957-140b-46ff-96aa-f0d57523183f\" (UID: \"cd37b957-140b-46ff-96aa-f0d57523183f\") "
Sep 17 17:30:12.462856 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.462829    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd37b957-140b-46ff-96aa-f0d57523183f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd37b957-140b-46ff-96aa-f0d57523183f" (UID: "cd37b957-140b-46ff-96aa-f0d57523183f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:30:12.462897 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.462856    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd37b957-140b-46ff-96aa-f0d57523183f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd37b957-140b-46ff-96aa-f0d57523183f" (UID: "cd37b957-140b-46ff-96aa-f0d57523183f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:30:12.462939 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.462898    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd37b957-140b-46ff-96aa-f0d57523183f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cd37b957-140b-46ff-96aa-f0d57523183f" (UID: "cd37b957-140b-46ff-96aa-f0d57523183f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:30:12.464640 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.464610    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd37b957-140b-46ff-96aa-f0d57523183f-kube-api-access-qkxx7" (OuterVolumeSpecName: "kube-api-access-qkxx7") pod "cd37b957-140b-46ff-96aa-f0d57523183f" (UID: "cd37b957-140b-46ff-96aa-f0d57523183f"). InnerVolumeSpecName "kube-api-access-qkxx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:30:12.464743 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.464663    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd37b957-140b-46ff-96aa-f0d57523183f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd37b957-140b-46ff-96aa-f0d57523183f" (UID: "cd37b957-140b-46ff-96aa-f0d57523183f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:30:12.563133 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.563108    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd37b957-140b-46ff-96aa-f0d57523183f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:30:12.563133 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.563129    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkxx7\" (UniqueName: \"kubernetes.io/projected/cd37b957-140b-46ff-96aa-f0d57523183f-kube-api-access-qkxx7\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:30:12.563283 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.563139    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd37b957-140b-46ff-96aa-f0d57523183f-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:30:12.563283 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.563148    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd37b957-140b-46ff-96aa-f0d57523183f-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:30:12.563283 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.563168    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd37b957-140b-46ff-96aa-f0d57523183f-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:30:12.855936 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.855904    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd37b957-140b-46ff-96aa-f0d57523183f" containerID="96aef3e004ff45ac29587318d2b321121600bd3b9d09cba6f4cae798c0137db3" exitCode=0
Sep 17 17:30:12.856110 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.855946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" event={"ID":"cd37b957-140b-46ff-96aa-f0d57523183f","Type":"ContainerDied","Data":"96aef3e004ff45ac29587318d2b321121600bd3b9d09cba6f4cae798c0137db3"}
Sep 17 17:30:12.856110 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.855959    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"
Sep 17 17:30:12.856110 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.855995    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq" event={"ID":"cd37b957-140b-46ff-96aa-f0d57523183f","Type":"ContainerDied","Data":"61d91d1dc1c0b9103e6d93d28f0f200984012cd046bf8c17970434d6b8efe835"}
Sep 17 17:30:12.856110 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.856021    2577 scope.go:117] "RemoveContainer" containerID="fe803f8987deb7ec0d95c0bc45f8218f00524298e34d88ece7c32752239b2344"
Sep 17 17:30:12.864804 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.864786    2577 scope.go:117] "RemoveContainer" containerID="96aef3e004ff45ac29587318d2b321121600bd3b9d09cba6f4cae798c0137db3"
Sep 17 17:30:12.872150 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.872120    2577 scope.go:117] "RemoveContainer" containerID="8fb3e468926c99bd89e86b798609832bf34395a731231c98bb8945a4efe6fd58"
Sep 17 17:30:12.878086 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.878064    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"]
Sep 17 17:30:12.879267 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.879231    2577 scope.go:117] "RemoveContainer" containerID="fe803f8987deb7ec0d95c0bc45f8218f00524298e34d88ece7c32752239b2344"
Sep 17 17:30:12.879518 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:30:12.879499    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe803f8987deb7ec0d95c0bc45f8218f00524298e34d88ece7c32752239b2344\": container with ID starting with fe803f8987deb7ec0d95c0bc45f8218f00524298e34d88ece7c32752239b2344 not found: ID does not exist" containerID="fe803f8987deb7ec0d95c0bc45f8218f00524298e34d88ece7c32752239b2344"
Sep 17 17:30:12.879574 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.879526    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe803f8987deb7ec0d95c0bc45f8218f00524298e34d88ece7c32752239b2344"} err="failed to get container status \"fe803f8987deb7ec0d95c0bc45f8218f00524298e34d88ece7c32752239b2344\": rpc error: code = NotFound desc = could not find container \"fe803f8987deb7ec0d95c0bc45f8218f00524298e34d88ece7c32752239b2344\": container with ID starting with fe803f8987deb7ec0d95c0bc45f8218f00524298e34d88ece7c32752239b2344 not found: ID does not exist"
Sep 17 17:30:12.879574 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.879543    2577 scope.go:117] "RemoveContainer" containerID="96aef3e004ff45ac29587318d2b321121600bd3b9d09cba6f4cae798c0137db3"
Sep 17 17:30:12.879779 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:30:12.879760    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96aef3e004ff45ac29587318d2b321121600bd3b9d09cba6f4cae798c0137db3\": container with ID starting with 96aef3e004ff45ac29587318d2b321121600bd3b9d09cba6f4cae798c0137db3 not found: ID does not exist" containerID="96aef3e004ff45ac29587318d2b321121600bd3b9d09cba6f4cae798c0137db3"
Sep 17 17:30:12.879846 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.879789    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96aef3e004ff45ac29587318d2b321121600bd3b9d09cba6f4cae798c0137db3"} err="failed to get container status \"96aef3e004ff45ac29587318d2b321121600bd3b9d09cba6f4cae798c0137db3\": rpc error: code = NotFound desc = could not find container \"96aef3e004ff45ac29587318d2b321121600bd3b9d09cba6f4cae798c0137db3\": container with ID starting with 96aef3e004ff45ac29587318d2b321121600bd3b9d09cba6f4cae798c0137db3 not found: ID does not exist"
Sep 17 17:30:12.879846 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.879811    2577 scope.go:117] "RemoveContainer" containerID="8fb3e468926c99bd89e86b798609832bf34395a731231c98bb8945a4efe6fd58"
Sep 17 17:30:12.880039 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:30:12.880023    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb3e468926c99bd89e86b798609832bf34395a731231c98bb8945a4efe6fd58\": container with ID starting with 8fb3e468926c99bd89e86b798609832bf34395a731231c98bb8945a4efe6fd58 not found: ID does not exist" containerID="8fb3e468926c99bd89e86b798609832bf34395a731231c98bb8945a4efe6fd58"
Sep 17 17:30:12.880092 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.880047    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb3e468926c99bd89e86b798609832bf34395a731231c98bb8945a4efe6fd58"} err="failed to get container status \"8fb3e468926c99bd89e86b798609832bf34395a731231c98bb8945a4efe6fd58\": rpc error: code = NotFound desc = could not find container \"8fb3e468926c99bd89e86b798609832bf34395a731231c98bb8945a4efe6fd58\": container with ID starting with 8fb3e468926c99bd89e86b798609832bf34395a731231c98bb8945a4efe6fd58 not found: ID does not exist"
Sep 17 17:30:12.882948 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:12.882928    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fttxq"]
Sep 17 17:30:13.917967 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.917933    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"]
Sep 17 17:30:13.918349 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.918289    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="storage-initializer"
Sep 17 17:30:13.918349 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.918301    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="storage-initializer"
Sep 17 17:30:13.918349 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.918314    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kube-rbac-proxy"
Sep 17 17:30:13.918349 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.918319    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kube-rbac-proxy"
Sep 17 17:30:13.918349 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.918327    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kserve-container"
Sep 17 17:30:13.918349 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.918334    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kserve-container"
Sep 17 17:30:13.918553 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.918382    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kserve-container"
Sep 17 17:30:13.918553 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.918391    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" containerName="kube-rbac-proxy"
Sep 17 17:30:13.922737 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.922716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:13.925134 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.925112    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:30:13.925280 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.925183    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 17 17:30:13.925342 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.925291    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 17 17:30:13.933510 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.933486    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"]
Sep 17 17:30:13.973315 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.973290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b64bbd-9b87-4b72-a156-457a80803dd6-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-mqdwq\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:13.973405 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.973334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2gfk\" (UniqueName: \"kubernetes.io/projected/35b64bbd-9b87-4b72-a156-457a80803dd6-kube-api-access-f2gfk\") pod \"isvc-pmml-runtime-predictor-885fc8777-mqdwq\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:13.973405 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.973364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b64bbd-9b87-4b72-a156-457a80803dd6-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-mqdwq\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:13.973405 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.973385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b64bbd-9b87-4b72-a156-457a80803dd6-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-mqdwq\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:13.973519 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:13.973420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b64bbd-9b87-4b72-a156-457a80803dd6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-mqdwq\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:14.073822 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:14.073795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b64bbd-9b87-4b72-a156-457a80803dd6-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-mqdwq\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:14.073915 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:14.073828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b64bbd-9b87-4b72-a156-457a80803dd6-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-mqdwq\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:14.073915 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:14.073849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b64bbd-9b87-4b72-a156-457a80803dd6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-mqdwq\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:14.073915 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:14.073898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b64bbd-9b87-4b72-a156-457a80803dd6-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-mqdwq\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:14.074048 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:14.073949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2gfk\" (UniqueName: \"kubernetes.io/projected/35b64bbd-9b87-4b72-a156-457a80803dd6-kube-api-access-f2gfk\") pod \"isvc-pmml-runtime-predictor-885fc8777-mqdwq\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:14.074196 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:14.074150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b64bbd-9b87-4b72-a156-457a80803dd6-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-mqdwq\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:14.074510 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:14.074492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b64bbd-9b87-4b72-a156-457a80803dd6-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-mqdwq\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:14.074577 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:14.074564    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b64bbd-9b87-4b72-a156-457a80803dd6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-mqdwq\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:14.076471 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:14.076448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b64bbd-9b87-4b72-a156-457a80803dd6-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-mqdwq\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:14.081470 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:14.081445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2gfk\" (UniqueName: \"kubernetes.io/projected/35b64bbd-9b87-4b72-a156-457a80803dd6-kube-api-access-f2gfk\") pod \"isvc-pmml-runtime-predictor-885fc8777-mqdwq\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:14.232618 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:14.232551    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:14.446855 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:14.446820    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd37b957-140b-46ff-96aa-f0d57523183f" path="/var/lib/kubelet/pods/cd37b957-140b-46ff-96aa-f0d57523183f/volumes"
Sep 17 17:30:14.507108 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:14.507044    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"]
Sep 17 17:30:14.509724 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:30:14.509697    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b64bbd_9b87_4b72_a156_457a80803dd6.slice/crio-6029bbc88c7cce5aeec9ad4eda02c2c8b3022ea43a202daf68d2069875075b7b WatchSource:0}: Error finding container 6029bbc88c7cce5aeec9ad4eda02c2c8b3022ea43a202daf68d2069875075b7b: Status 404 returned error can't find the container with id 6029bbc88c7cce5aeec9ad4eda02c2c8b3022ea43a202daf68d2069875075b7b
Sep 17 17:30:14.868345 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:14.868309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" event={"ID":"35b64bbd-9b87-4b72-a156-457a80803dd6","Type":"ContainerStarted","Data":"1965a9899f616d050c58edc46dd243530ee30268cf70b4c5acd0db4b0a789ff9"}
Sep 17 17:30:14.868514 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:14.868366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" event={"ID":"35b64bbd-9b87-4b72-a156-457a80803dd6","Type":"ContainerStarted","Data":"6029bbc88c7cce5aeec9ad4eda02c2c8b3022ea43a202daf68d2069875075b7b"}
Sep 17 17:30:18.591267 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:18.591233    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 17:30:18.887875 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:18.887798    2577 generic.go:356] "Generic (PLEG): container finished" podID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerID="1965a9899f616d050c58edc46dd243530ee30268cf70b4c5acd0db4b0a789ff9" exitCode=0
Sep 17 17:30:18.888011 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:18.887876    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" event={"ID":"35b64bbd-9b87-4b72-a156-457a80803dd6","Type":"ContainerDied","Data":"1965a9899f616d050c58edc46dd243530ee30268cf70b4c5acd0db4b0a789ff9"}
Sep 17 17:30:19.894217 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:19.894182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" event={"ID":"35b64bbd-9b87-4b72-a156-457a80803dd6","Type":"ContainerStarted","Data":"ee7df4d09c3e0590f72464a2299d731a06084630d4efea8076edbe996482ea0a"}
Sep 17 17:30:19.894717 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:19.894245    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" event={"ID":"35b64bbd-9b87-4b72-a156-457a80803dd6","Type":"ContainerStarted","Data":"5ec29b7ed2bcd6ea603135fce3c8d4c41d6f7a3eaa63750aa1cb1d8b36e35910"}
Sep 17 17:30:19.894717 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:19.894428    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:19.914447 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:19.914406    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" podStartSLOduration=3.196947195 podStartE2EDuration="6.914392558s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.717445363s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:30:19.912349692 +0000 UTC m=+2937.937257981" watchObservedRunningTime="2026-09-17 17:30:19.914392558 +0000 UTC m=+2937.939300846"
Sep 17 17:30:20.899104 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:20.899069    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:20.900354 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:20.900324    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 17:30:21.903263 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:21.903220    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 17:30:26.907476 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:26.907442    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:30:26.907958 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:26.907937    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 17:30:28.591313 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:28.591284    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:30:36.908304 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:36.908255    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 17:30:38.625338 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:38.625308    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"]
Sep 17 17:30:38.625824 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:38.625537    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kserve-container" containerID="cri-o://032907a87f3e3c554a998f06047e88ee877dd89ad421de4bf8b2a9f920846a8a" gracePeriod=30
Sep 17 17:30:38.625824 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:38.625577    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kube-rbac-proxy" containerID="cri-o://38d55c1880493b237377f38ca6d94e359ac12f60b65a2c41a39a4b48bff76628" gracePeriod=30
Sep 17 17:30:38.980186 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:38.980082    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerID="38d55c1880493b237377f38ca6d94e359ac12f60b65a2c41a39a4b48bff76628" exitCode=2
Sep 17 17:30:38.980340 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:38.980177    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" event={"ID":"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b","Type":"ContainerDied","Data":"38d55c1880493b237377f38ca6d94e359ac12f60b65a2c41a39a4b48bff76628"}
Sep 17 17:30:42.674674 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.674648    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:30:42.780980 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.780914    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-cabundle-cert\") pod \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") "
Sep 17 17:30:42.780980 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.780957    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s686n\" (UniqueName: \"kubernetes.io/projected/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kube-api-access-s686n\") pod \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") "
Sep 17 17:30:42.781148 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.780992    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kube-rbac-proxy-sar-config\") pod \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") "
Sep 17 17:30:42.781148 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.781017    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kserve-provision-location\") pod \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") "
Sep 17 17:30:42.781148 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.781047    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-proxy-tls\") pod \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\" (UID: \"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b\") "
Sep 17 17:30:42.781384 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.781361    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" (UID: "eb850cdc-5a9a-4d34-8a92-d47c7a9b576b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:30:42.781455 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.781403    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" (UID: "eb850cdc-5a9a-4d34-8a92-d47c7a9b576b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:30:42.781511 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.781461    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" (UID: "eb850cdc-5a9a-4d34-8a92-d47c7a9b576b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:30:42.783082 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.783055    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" (UID: "eb850cdc-5a9a-4d34-8a92-d47c7a9b576b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:30:42.783207 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.783185    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kube-api-access-s686n" (OuterVolumeSpecName: "kube-api-access-s686n") pod "eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" (UID: "eb850cdc-5a9a-4d34-8a92-d47c7a9b576b"). InnerVolumeSpecName "kube-api-access-s686n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:30:42.882471 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.882443    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:30:42.882471 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.882467    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:30:42.882605 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.882478    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:30:42.882605 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.882487    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s686n\" (UniqueName: \"kubernetes.io/projected/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kube-api-access-s686n\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:30:42.882605 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:42.882496    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:30:43.000571 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:43.000542    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerID="032907a87f3e3c554a998f06047e88ee877dd89ad421de4bf8b2a9f920846a8a" exitCode=0
Sep 17 17:30:43.000706 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:43.000583    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" event={"ID":"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b","Type":"ContainerDied","Data":"032907a87f3e3c554a998f06047e88ee877dd89ad421de4bf8b2a9f920846a8a"}
Sep 17 17:30:43.000706 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:43.000611    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"
Sep 17 17:30:43.000706 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:43.000628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8" event={"ID":"eb850cdc-5a9a-4d34-8a92-d47c7a9b576b","Type":"ContainerDied","Data":"1c56ad99ce999c2e5ea62bf61cc6567626c2890427508344077fce568f5e2dcb"}
Sep 17 17:30:43.000706 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:43.000655    2577 scope.go:117] "RemoveContainer" containerID="38d55c1880493b237377f38ca6d94e359ac12f60b65a2c41a39a4b48bff76628"
Sep 17 17:30:43.010597 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:43.010574    2577 scope.go:117] "RemoveContainer" containerID="032907a87f3e3c554a998f06047e88ee877dd89ad421de4bf8b2a9f920846a8a"
Sep 17 17:30:43.018292 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:43.018275    2577 scope.go:117] "RemoveContainer" containerID="c5a1d9f7aa158addcfb30503f3724f9c392adc251852475ab67937f62f67707c"
Sep 17 17:30:43.024644 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:43.024622    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"]
Sep 17 17:30:43.025324 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:43.025305    2577 scope.go:117] "RemoveContainer" containerID="38d55c1880493b237377f38ca6d94e359ac12f60b65a2c41a39a4b48bff76628"
Sep 17 17:30:43.025595 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:30:43.025577    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d55c1880493b237377f38ca6d94e359ac12f60b65a2c41a39a4b48bff76628\": container with ID starting with 38d55c1880493b237377f38ca6d94e359ac12f60b65a2c41a39a4b48bff76628 not found: ID does not exist" containerID="38d55c1880493b237377f38ca6d94e359ac12f60b65a2c41a39a4b48bff76628"
Sep 17 17:30:43.025642 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:43.025603    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d55c1880493b237377f38ca6d94e359ac12f60b65a2c41a39a4b48bff76628"} err="failed to get container status \"38d55c1880493b237377f38ca6d94e359ac12f60b65a2c41a39a4b48bff76628\": rpc error: code = NotFound desc = could not find container \"38d55c1880493b237377f38ca6d94e359ac12f60b65a2c41a39a4b48bff76628\": container with ID starting with 38d55c1880493b237377f38ca6d94e359ac12f60b65a2c41a39a4b48bff76628 not found: ID does not exist"
Sep 17 17:30:43.025642 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:43.025620    2577 scope.go:117] "RemoveContainer" containerID="032907a87f3e3c554a998f06047e88ee877dd89ad421de4bf8b2a9f920846a8a"
Sep 17 17:30:43.025847 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:30:43.025831    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032907a87f3e3c554a998f06047e88ee877dd89ad421de4bf8b2a9f920846a8a\": container with ID starting with 032907a87f3e3c554a998f06047e88ee877dd89ad421de4bf8b2a9f920846a8a not found: ID does not exist" containerID="032907a87f3e3c554a998f06047e88ee877dd89ad421de4bf8b2a9f920846a8a"
Sep 17 17:30:43.025892 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:43.025853    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032907a87f3e3c554a998f06047e88ee877dd89ad421de4bf8b2a9f920846a8a"} err="failed to get container status \"032907a87f3e3c554a998f06047e88ee877dd89ad421de4bf8b2a9f920846a8a\": rpc error: code = NotFound desc = could not find container \"032907a87f3e3c554a998f06047e88ee877dd89ad421de4bf8b2a9f920846a8a\": container with ID starting with 032907a87f3e3c554a998f06047e88ee877dd89ad421de4bf8b2a9f920846a8a not found: ID does not exist"
Sep 17 17:30:43.025892 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:43.025867    2577 scope.go:117] "RemoveContainer" containerID="c5a1d9f7aa158addcfb30503f3724f9c392adc251852475ab67937f62f67707c"
Sep 17 17:30:43.026068 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:30:43.026053    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a1d9f7aa158addcfb30503f3724f9c392adc251852475ab67937f62f67707c\": container with ID starting with c5a1d9f7aa158addcfb30503f3724f9c392adc251852475ab67937f62f67707c not found: ID does not exist" containerID="c5a1d9f7aa158addcfb30503f3724f9c392adc251852475ab67937f62f67707c"
Sep 17 17:30:43.026109 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:43.026071    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a1d9f7aa158addcfb30503f3724f9c392adc251852475ab67937f62f67707c"} err="failed to get container status \"c5a1d9f7aa158addcfb30503f3724f9c392adc251852475ab67937f62f67707c\": rpc error: code = NotFound desc = could not find container \"c5a1d9f7aa158addcfb30503f3724f9c392adc251852475ab67937f62f67707c\": container with ID starting with c5a1d9f7aa158addcfb30503f3724f9c392adc251852475ab67937f62f67707c not found: ID does not exist"
Sep 17 17:30:43.030173 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:43.030142    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c7fd78fd-gtxg8"]
Sep 17 17:30:44.449823 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:44.449777    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" path="/var/lib/kubelet/pods/eb850cdc-5a9a-4d34-8a92-d47c7a9b576b/volumes"
Sep 17 17:30:46.908301 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:46.908216    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 17:30:50.866672 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.866637    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"]
Sep 17 17:30:50.867103 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.866944    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="storage-initializer"
Sep 17 17:30:50.867103 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.866954    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="storage-initializer"
Sep 17 17:30:50.867103 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.866969    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kube-rbac-proxy"
Sep 17 17:30:50.867103 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.866976    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kube-rbac-proxy"
Sep 17 17:30:50.867103 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.866989    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kserve-container"
Sep 17 17:30:50.867103 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.866995    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kserve-container"
Sep 17 17:30:50.867103 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.867050    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kube-rbac-proxy"
Sep 17 17:30:50.867103 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.867057    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb850cdc-5a9a-4d34-8a92-d47c7a9b576b" containerName="kserve-container"
Sep 17 17:30:50.871698 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.871680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:50.873952 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.873929    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-442a34-predictor-serving-cert\""
Sep 17 17:30:50.874043 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.874011    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-442a34-kube-rbac-proxy-sar-config\""
Sep 17 17:30:50.879224 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.879204    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"]
Sep 17 17:30:50.942334 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.942305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/378254bf-ac71-46fe-8afa-aa28bf90634d-proxy-tls\") pod \"isvc-primary-442a34-predictor-668f45cfcc-qdwrg\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") " pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:50.942479 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.942343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/378254bf-ac71-46fe-8afa-aa28bf90634d-cabundle-cert\") pod \"isvc-primary-442a34-predictor-668f45cfcc-qdwrg\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") " pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:50.942479 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.942376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj2xp\" (UniqueName: \"kubernetes.io/projected/378254bf-ac71-46fe-8afa-aa28bf90634d-kube-api-access-sj2xp\") pod \"isvc-primary-442a34-predictor-668f45cfcc-qdwrg\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") " pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:50.942479 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.942432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/378254bf-ac71-46fe-8afa-aa28bf90634d-kserve-provision-location\") pod \"isvc-primary-442a34-predictor-668f45cfcc-qdwrg\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") " pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:50.942590 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:50.942483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/378254bf-ac71-46fe-8afa-aa28bf90634d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-442a34-predictor-668f45cfcc-qdwrg\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") " pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:51.042981 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:51.042950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/378254bf-ac71-46fe-8afa-aa28bf90634d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-442a34-predictor-668f45cfcc-qdwrg\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") " pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:51.043152 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:51.043004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/378254bf-ac71-46fe-8afa-aa28bf90634d-proxy-tls\") pod \"isvc-primary-442a34-predictor-668f45cfcc-qdwrg\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") " pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:51.043152 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:51.043046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/378254bf-ac71-46fe-8afa-aa28bf90634d-cabundle-cert\") pod \"isvc-primary-442a34-predictor-668f45cfcc-qdwrg\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") " pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:51.043152 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:51.043080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj2xp\" (UniqueName: \"kubernetes.io/projected/378254bf-ac71-46fe-8afa-aa28bf90634d-kube-api-access-sj2xp\") pod \"isvc-primary-442a34-predictor-668f45cfcc-qdwrg\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") " pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:51.043152 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:51.043113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/378254bf-ac71-46fe-8afa-aa28bf90634d-kserve-provision-location\") pod \"isvc-primary-442a34-predictor-668f45cfcc-qdwrg\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") " pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:51.043587 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:51.043562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/378254bf-ac71-46fe-8afa-aa28bf90634d-kserve-provision-location\") pod \"isvc-primary-442a34-predictor-668f45cfcc-qdwrg\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") " pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:51.043768 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:51.043749    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/378254bf-ac71-46fe-8afa-aa28bf90634d-cabundle-cert\") pod \"isvc-primary-442a34-predictor-668f45cfcc-qdwrg\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") " pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:51.043815 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:51.043763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/378254bf-ac71-46fe-8afa-aa28bf90634d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-442a34-predictor-668f45cfcc-qdwrg\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") " pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:51.045511 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:51.045495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/378254bf-ac71-46fe-8afa-aa28bf90634d-proxy-tls\") pod \"isvc-primary-442a34-predictor-668f45cfcc-qdwrg\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") " pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:51.050382 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:51.050363    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj2xp\" (UniqueName: \"kubernetes.io/projected/378254bf-ac71-46fe-8afa-aa28bf90634d-kube-api-access-sj2xp\") pod \"isvc-primary-442a34-predictor-668f45cfcc-qdwrg\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") " pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:51.182409 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:51.182344    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:51.457750 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:51.457723    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"]
Sep 17 17:30:51.460109 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:30:51.460082    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378254bf_ac71_46fe_8afa_aa28bf90634d.slice/crio-e715fc992da4d7ec5e4127bfe960834a6946a8b2f108c57cfffb99736d0a6e1b WatchSource:0}: Error finding container e715fc992da4d7ec5e4127bfe960834a6946a8b2f108c57cfffb99736d0a6e1b: Status 404 returned error can't find the container with id e715fc992da4d7ec5e4127bfe960834a6946a8b2f108c57cfffb99736d0a6e1b
Sep 17 17:30:52.048874 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:52.048830    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" event={"ID":"378254bf-ac71-46fe-8afa-aa28bf90634d","Type":"ContainerStarted","Data":"6d919b99af026bce8f7a3615eab6507cb67a134028850243375b4ee98143f75c"}
Sep 17 17:30:52.049262 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:52.048916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" event={"ID":"378254bf-ac71-46fe-8afa-aa28bf90634d","Type":"ContainerStarted","Data":"e715fc992da4d7ec5e4127bfe960834a6946a8b2f108c57cfffb99736d0a6e1b"}
Sep 17 17:30:56.068062 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:56.068030    2577 generic.go:356] "Generic (PLEG): container finished" podID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerID="6d919b99af026bce8f7a3615eab6507cb67a134028850243375b4ee98143f75c" exitCode=0
Sep 17 17:30:56.068449 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:56.068113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" event={"ID":"378254bf-ac71-46fe-8afa-aa28bf90634d","Type":"ContainerDied","Data":"6d919b99af026bce8f7a3615eab6507cb67a134028850243375b4ee98143f75c"}
Sep 17 17:30:56.907957 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:56.907911    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 17:30:57.074570 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:57.074538    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" event={"ID":"378254bf-ac71-46fe-8afa-aa28bf90634d","Type":"ContainerStarted","Data":"2a608982925086a5b7c2bdb0de56e0d4e0316a66bc02bfc38c3123e639c495ed"}
Sep 17 17:30:57.074945 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:57.074599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" event={"ID":"378254bf-ac71-46fe-8afa-aa28bf90634d","Type":"ContainerStarted","Data":"9763706cf3289e4daecb5b8edac6164e4be747830c1e13806981cd2353d26612"}
Sep 17 17:30:57.074945 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:57.074869    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:57.075027 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:57.075003    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:30:57.076299 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:57.076275    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 17:30:57.095910 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:57.095872    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" podStartSLOduration=3.285811163 podStartE2EDuration="7.095859436s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.810048273s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:30:57.094400119 +0000 UTC m=+2975.119308418" watchObservedRunningTime="2026-09-17 17:30:57.095859436 +0000 UTC m=+2975.120767724"
Sep 17 17:30:58.079714 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:30:58.079671    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 17:31:03.084567 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:03.084535    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:31:03.084990 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:03.084954    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 17:31:06.908063 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:06.908015    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 17:31:13.085255 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:13.085213    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 17:31:16.907943 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:16.907898    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 17:31:22.642765 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:22.642733    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 17:31:22.649708 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:22.649689    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 17:31:23.085433 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:23.085393    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 17:31:26.908835 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:26.908786    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 17:31:33.085796 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:33.085757    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 17:31:36.908758 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:36.908722    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 17:31:43.085375 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:43.085335    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 17:31:46.909311 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:46.909284    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:31:53.085787 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:53.085744    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 17:31:55.529422 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:55.529392    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"]
Sep 17 17:31:55.529933 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:55.529624    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kserve-container" containerID="cri-o://5ec29b7ed2bcd6ea603135fce3c8d4c41d6f7a3eaa63750aa1cb1d8b36e35910" gracePeriod=30
Sep 17 17:31:55.529933 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:55.529672    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kube-rbac-proxy" containerID="cri-o://ee7df4d09c3e0590f72464a2299d731a06084630d4efea8076edbe996482ea0a" gracePeriod=30
Sep 17 17:31:56.343699 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:56.343668    2577 generic.go:356] "Generic (PLEG): container finished" podID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerID="ee7df4d09c3e0590f72464a2299d731a06084630d4efea8076edbe996482ea0a" exitCode=2
Sep 17 17:31:56.343869 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:56.343746    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" event={"ID":"35b64bbd-9b87-4b72-a156-457a80803dd6","Type":"ContainerDied","Data":"ee7df4d09c3e0590f72464a2299d731a06084630d4efea8076edbe996482ea0a"}
Sep 17 17:31:56.904266 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:56.904216    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Sep 17 17:31:56.908506 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:56.908476    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 17:31:59.359562 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.359524    2577 generic.go:356] "Generic (PLEG): container finished" podID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerID="5ec29b7ed2bcd6ea603135fce3c8d4c41d6f7a3eaa63750aa1cb1d8b36e35910" exitCode=0
Sep 17 17:31:59.359857 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.359580    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" event={"ID":"35b64bbd-9b87-4b72-a156-457a80803dd6","Type":"ContainerDied","Data":"5ec29b7ed2bcd6ea603135fce3c8d4c41d6f7a3eaa63750aa1cb1d8b36e35910"}
Sep 17 17:31:59.480695 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.480673    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:31:59.567444 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.567385    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b64bbd-9b87-4b72-a156-457a80803dd6-kserve-provision-location\") pod \"35b64bbd-9b87-4b72-a156-457a80803dd6\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") "
Sep 17 17:31:59.567568 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.567451    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b64bbd-9b87-4b72-a156-457a80803dd6-cabundle-cert\") pod \"35b64bbd-9b87-4b72-a156-457a80803dd6\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") "
Sep 17 17:31:59.567568 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.567476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2gfk\" (UniqueName: \"kubernetes.io/projected/35b64bbd-9b87-4b72-a156-457a80803dd6-kube-api-access-f2gfk\") pod \"35b64bbd-9b87-4b72-a156-457a80803dd6\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") "
Sep 17 17:31:59.567568 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.567498    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b64bbd-9b87-4b72-a156-457a80803dd6-proxy-tls\") pod \"35b64bbd-9b87-4b72-a156-457a80803dd6\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") "
Sep 17 17:31:59.567568 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.567514    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b64bbd-9b87-4b72-a156-457a80803dd6-kube-rbac-proxy-sar-config\") pod \"35b64bbd-9b87-4b72-a156-457a80803dd6\" (UID: \"35b64bbd-9b87-4b72-a156-457a80803dd6\") "
Sep 17 17:31:59.567773 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.567745    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b64bbd-9b87-4b72-a156-457a80803dd6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35b64bbd-9b87-4b72-a156-457a80803dd6" (UID: "35b64bbd-9b87-4b72-a156-457a80803dd6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:31:59.567862 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.567842    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b64bbd-9b87-4b72-a156-457a80803dd6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "35b64bbd-9b87-4b72-a156-457a80803dd6" (UID: "35b64bbd-9b87-4b72-a156-457a80803dd6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:31:59.567914 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.567893    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b64bbd-9b87-4b72-a156-457a80803dd6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "35b64bbd-9b87-4b72-a156-457a80803dd6" (UID: "35b64bbd-9b87-4b72-a156-457a80803dd6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:31:59.569556 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.569533    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b64bbd-9b87-4b72-a156-457a80803dd6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35b64bbd-9b87-4b72-a156-457a80803dd6" (UID: "35b64bbd-9b87-4b72-a156-457a80803dd6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:31:59.569626 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.569590    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b64bbd-9b87-4b72-a156-457a80803dd6-kube-api-access-f2gfk" (OuterVolumeSpecName: "kube-api-access-f2gfk") pod "35b64bbd-9b87-4b72-a156-457a80803dd6" (UID: "35b64bbd-9b87-4b72-a156-457a80803dd6"). InnerVolumeSpecName "kube-api-access-f2gfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:31:59.668887 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.668864    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b64bbd-9b87-4b72-a156-457a80803dd6-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:31:59.668887 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.668883    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2gfk\" (UniqueName: \"kubernetes.io/projected/35b64bbd-9b87-4b72-a156-457a80803dd6-kube-api-access-f2gfk\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:31:59.668887 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.668893    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b64bbd-9b87-4b72-a156-457a80803dd6-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:31:59.669079 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.668902    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b64bbd-9b87-4b72-a156-457a80803dd6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:31:59.669079 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:31:59.668912    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b64bbd-9b87-4b72-a156-457a80803dd6-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:32:00.366368 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.366335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq" event={"ID":"35b64bbd-9b87-4b72-a156-457a80803dd6","Type":"ContainerDied","Data":"6029bbc88c7cce5aeec9ad4eda02c2c8b3022ea43a202daf68d2069875075b7b"}
Sep 17 17:32:00.366860 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.366383    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"
Sep 17 17:32:00.366860 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.366401    2577 scope.go:117] "RemoveContainer" containerID="ee7df4d09c3e0590f72464a2299d731a06084630d4efea8076edbe996482ea0a"
Sep 17 17:32:00.374949 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.374930    2577 scope.go:117] "RemoveContainer" containerID="5ec29b7ed2bcd6ea603135fce3c8d4c41d6f7a3eaa63750aa1cb1d8b36e35910"
Sep 17 17:32:00.382309 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.382292    2577 scope.go:117] "RemoveContainer" containerID="1965a9899f616d050c58edc46dd243530ee30268cf70b4c5acd0db4b0a789ff9"
Sep 17 17:32:00.389249 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.389227    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"]
Sep 17 17:32:00.391535 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.391517    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-mqdwq"]
Sep 17 17:32:00.447232 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.447202    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" path="/var/lib/kubelet/pods/35b64bbd-9b87-4b72-a156-457a80803dd6/volumes"
Sep 17 17:32:00.727947 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.727876    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"]
Sep 17 17:32:00.728216 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.728198    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kube-rbac-proxy"
Sep 17 17:32:00.728289 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.728218    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kube-rbac-proxy"
Sep 17 17:32:00.728289 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.728229    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kserve-container"
Sep 17 17:32:00.728289 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.728235    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kserve-container"
Sep 17 17:32:00.728289 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.728247    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="storage-initializer"
Sep 17 17:32:00.728289 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.728253    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="storage-initializer"
Sep 17 17:32:00.728450 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.728312    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kserve-container"
Sep 17 17:32:00.728450 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.728322    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b64bbd-9b87-4b72-a156-457a80803dd6" containerName="kube-rbac-proxy"
Sep 17 17:32:00.732796 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.732778    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:00.735093 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.735067    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 17 17:32:00.735236 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.735121    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 17:32:00.743136 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.743116    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"]
Sep 17 17:32:00.778044 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.778022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdqb\" (UniqueName: \"kubernetes.io/projected/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kube-api-access-sxdqb\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:00.778150 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.778067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:00.778150 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.778113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57bd54f1-3087-4bc9-8696-9d8c0c88555a-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:00.778150 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.778142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:00.778303 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.778177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57bd54f1-3087-4bc9-8696-9d8c0c88555a-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:00.878858 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.878831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:00.878987 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.878877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57bd54f1-3087-4bc9-8696-9d8c0c88555a-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:00.878987 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.878908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:00.878987 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.878932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57bd54f1-3087-4bc9-8696-9d8c0c88555a-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:00.878987 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.878983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdqb\" (UniqueName: \"kubernetes.io/projected/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kube-api-access-sxdqb\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:00.879398 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.879376    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:00.879523 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.879506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:00.879639 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.879621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57bd54f1-3087-4bc9-8696-9d8c0c88555a-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:00.881417 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.881400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57bd54f1-3087-4bc9-8696-9d8c0c88555a-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:00.886398 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:00.886380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdqb\" (UniqueName: \"kubernetes.io/projected/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kube-api-access-sxdqb\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:01.043279 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:01.043252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:01.320931 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:01.320909    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"]
Sep 17 17:32:01.323137 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:32:01.323107    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57bd54f1_3087_4bc9_8696_9d8c0c88555a.slice/crio-303cfcb9e7543479f786f187a6d3a227a4ae1df3aa6e365687b85ddf506decb7 WatchSource:0}: Error finding container 303cfcb9e7543479f786f187a6d3a227a4ae1df3aa6e365687b85ddf506decb7: Status 404 returned error can't find the container with id 303cfcb9e7543479f786f187a6d3a227a4ae1df3aa6e365687b85ddf506decb7
Sep 17 17:32:01.378875 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:01.378842    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" event={"ID":"57bd54f1-3087-4bc9-8696-9d8c0c88555a","Type":"ContainerStarted","Data":"303cfcb9e7543479f786f187a6d3a227a4ae1df3aa6e365687b85ddf506decb7"}
Sep 17 17:32:02.385576 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:02.385537    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" event={"ID":"57bd54f1-3087-4bc9-8696-9d8c0c88555a","Type":"ContainerStarted","Data":"5ba8b48d206d86bbb22abeab8f21cb741b8ae270ad2c706a4a43c4e6f742dd5d"}
Sep 17 17:32:03.085668 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:03.085626    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 17:32:05.400150 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:05.400118    2577 generic.go:356] "Generic (PLEG): container finished" podID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerID="5ba8b48d206d86bbb22abeab8f21cb741b8ae270ad2c706a4a43c4e6f742dd5d" exitCode=0
Sep 17 17:32:05.400477 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:05.400190    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" event={"ID":"57bd54f1-3087-4bc9-8696-9d8c0c88555a","Type":"ContainerDied","Data":"5ba8b48d206d86bbb22abeab8f21cb741b8ae270ad2c706a4a43c4e6f742dd5d"}
Sep 17 17:32:06.407176 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:06.407132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" event={"ID":"57bd54f1-3087-4bc9-8696-9d8c0c88555a","Type":"ContainerStarted","Data":"61756d49c1a30880dedc405ee0b3823b470a2594fdf674510d32c3bdf3c14b4b"}
Sep 17 17:32:06.407545 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:06.407231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" event={"ID":"57bd54f1-3087-4bc9-8696-9d8c0c88555a","Type":"ContainerStarted","Data":"2e5ff8cf3830ef54d9e83b4e7f20c87b4b6ef5ba2a49efd95af057fc6913ebd3"}
Sep 17 17:32:06.407545 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:06.407520    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:06.425886 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:06.425841    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" podStartSLOduration=2.58468502 podStartE2EDuration="6.425828151s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.841143131s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:32:06.425069796 +0000 UTC m=+3044.449978109" watchObservedRunningTime="2026-09-17 17:32:06.425828151 +0000 UTC m=+3044.450736438"
Sep 17 17:32:07.418035 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:07.418004    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:07.419379 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:07.419352    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 17:32:08.422113 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:08.422066    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 17:32:13.085886 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:13.085853    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:32:13.426356 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:13.426281    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:32:13.426805 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:13.426776    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 17:32:23.427475 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:23.427439    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 17:32:33.427617 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:33.427576    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 17:32:43.426812 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:43.426772    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 17:32:53.426754 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:53.426717    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 17:32:54.071562 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:54.071528    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"]
Sep 17 17:32:54.071842 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:54.071813    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kserve-container" containerID="cri-o://9763706cf3289e4daecb5b8edac6164e4be747830c1e13806981cd2353d26612" gracePeriod=30
Sep 17 17:32:54.071951 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:54.071873    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kube-rbac-proxy" containerID="cri-o://2a608982925086a5b7c2bdb0de56e0d4e0316a66bc02bfc38c3123e639c495ed" gracePeriod=30
Sep 17 17:32:54.623373 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:54.623342    2577 generic.go:356] "Generic (PLEG): container finished" podID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerID="2a608982925086a5b7c2bdb0de56e0d4e0316a66bc02bfc38c3123e639c495ed" exitCode=2
Sep 17 17:32:54.623743 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:54.623379    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" event={"ID":"378254bf-ac71-46fe-8afa-aa28bf90634d","Type":"ContainerDied","Data":"2a608982925086a5b7c2bdb0de56e0d4e0316a66bc02bfc38c3123e639c495ed"}
Sep 17 17:32:58.079825 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.079778    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused"
Sep 17 17:32:58.439440 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.439417    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:32:58.601195 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.601105    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/378254bf-ac71-46fe-8afa-aa28bf90634d-cabundle-cert\") pod \"378254bf-ac71-46fe-8afa-aa28bf90634d\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") "
Sep 17 17:32:58.601195 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.601187    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj2xp\" (UniqueName: \"kubernetes.io/projected/378254bf-ac71-46fe-8afa-aa28bf90634d-kube-api-access-sj2xp\") pod \"378254bf-ac71-46fe-8afa-aa28bf90634d\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") "
Sep 17 17:32:58.601386 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.601211    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/378254bf-ac71-46fe-8afa-aa28bf90634d-kube-rbac-proxy-sar-config\") pod \"378254bf-ac71-46fe-8afa-aa28bf90634d\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") "
Sep 17 17:32:58.601386 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.601242    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/378254bf-ac71-46fe-8afa-aa28bf90634d-kserve-provision-location\") pod \"378254bf-ac71-46fe-8afa-aa28bf90634d\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") "
Sep 17 17:32:58.601386 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.601275    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/378254bf-ac71-46fe-8afa-aa28bf90634d-proxy-tls\") pod \"378254bf-ac71-46fe-8afa-aa28bf90634d\" (UID: \"378254bf-ac71-46fe-8afa-aa28bf90634d\") "
Sep 17 17:32:58.601548 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.601532    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378254bf-ac71-46fe-8afa-aa28bf90634d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "378254bf-ac71-46fe-8afa-aa28bf90634d" (UID: "378254bf-ac71-46fe-8afa-aa28bf90634d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:32:58.601609 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.601578    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378254bf-ac71-46fe-8afa-aa28bf90634d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "378254bf-ac71-46fe-8afa-aa28bf90634d" (UID: "378254bf-ac71-46fe-8afa-aa28bf90634d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:32:58.601668 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.601611    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378254bf-ac71-46fe-8afa-aa28bf90634d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "378254bf-ac71-46fe-8afa-aa28bf90634d" (UID: "378254bf-ac71-46fe-8afa-aa28bf90634d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:32:58.603273 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.603249    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378254bf-ac71-46fe-8afa-aa28bf90634d-kube-api-access-sj2xp" (OuterVolumeSpecName: "kube-api-access-sj2xp") pod "378254bf-ac71-46fe-8afa-aa28bf90634d" (UID: "378254bf-ac71-46fe-8afa-aa28bf90634d"). InnerVolumeSpecName "kube-api-access-sj2xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:32:58.603350 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.603253    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/378254bf-ac71-46fe-8afa-aa28bf90634d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "378254bf-ac71-46fe-8afa-aa28bf90634d" (UID: "378254bf-ac71-46fe-8afa-aa28bf90634d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:32:58.641198 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.641150    2577 generic.go:356] "Generic (PLEG): container finished" podID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerID="9763706cf3289e4daecb5b8edac6164e4be747830c1e13806981cd2353d26612" exitCode=0
Sep 17 17:32:58.641198 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.641188    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" event={"ID":"378254bf-ac71-46fe-8afa-aa28bf90634d","Type":"ContainerDied","Data":"9763706cf3289e4daecb5b8edac6164e4be747830c1e13806981cd2353d26612"}
Sep 17 17:32:58.641319 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.641215    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"
Sep 17 17:32:58.641319 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.641244    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg" event={"ID":"378254bf-ac71-46fe-8afa-aa28bf90634d","Type":"ContainerDied","Data":"e715fc992da4d7ec5e4127bfe960834a6946a8b2f108c57cfffb99736d0a6e1b"}
Sep 17 17:32:58.641319 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.641272    2577 scope.go:117] "RemoveContainer" containerID="2a608982925086a5b7c2bdb0de56e0d4e0316a66bc02bfc38c3123e639c495ed"
Sep 17 17:32:58.649446 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.649430    2577 scope.go:117] "RemoveContainer" containerID="9763706cf3289e4daecb5b8edac6164e4be747830c1e13806981cd2353d26612"
Sep 17 17:32:58.656728 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.656712    2577 scope.go:117] "RemoveContainer" containerID="6d919b99af026bce8f7a3615eab6507cb67a134028850243375b4ee98143f75c"
Sep 17 17:32:58.663203 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.663181    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"]
Sep 17 17:32:58.663987 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.663966    2577 scope.go:117] "RemoveContainer" containerID="2a608982925086a5b7c2bdb0de56e0d4e0316a66bc02bfc38c3123e639c495ed"
Sep 17 17:32:58.664267 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:32:58.664248    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a608982925086a5b7c2bdb0de56e0d4e0316a66bc02bfc38c3123e639c495ed\": container with ID starting with 2a608982925086a5b7c2bdb0de56e0d4e0316a66bc02bfc38c3123e639c495ed not found: ID does not exist" containerID="2a608982925086a5b7c2bdb0de56e0d4e0316a66bc02bfc38c3123e639c495ed"
Sep 17 17:32:58.664324 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.664276    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a608982925086a5b7c2bdb0de56e0d4e0316a66bc02bfc38c3123e639c495ed"} err="failed to get container status \"2a608982925086a5b7c2bdb0de56e0d4e0316a66bc02bfc38c3123e639c495ed\": rpc error: code = NotFound desc = could not find container \"2a608982925086a5b7c2bdb0de56e0d4e0316a66bc02bfc38c3123e639c495ed\": container with ID starting with 2a608982925086a5b7c2bdb0de56e0d4e0316a66bc02bfc38c3123e639c495ed not found: ID does not exist"
Sep 17 17:32:58.664324 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.664293    2577 scope.go:117] "RemoveContainer" containerID="9763706cf3289e4daecb5b8edac6164e4be747830c1e13806981cd2353d26612"
Sep 17 17:32:58.664565 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:32:58.664535    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9763706cf3289e4daecb5b8edac6164e4be747830c1e13806981cd2353d26612\": container with ID starting with 9763706cf3289e4daecb5b8edac6164e4be747830c1e13806981cd2353d26612 not found: ID does not exist" containerID="9763706cf3289e4daecb5b8edac6164e4be747830c1e13806981cd2353d26612"
Sep 17 17:32:58.664638 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.664572    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9763706cf3289e4daecb5b8edac6164e4be747830c1e13806981cd2353d26612"} err="failed to get container status \"9763706cf3289e4daecb5b8edac6164e4be747830c1e13806981cd2353d26612\": rpc error: code = NotFound desc = could not find container \"9763706cf3289e4daecb5b8edac6164e4be747830c1e13806981cd2353d26612\": container with ID starting with 9763706cf3289e4daecb5b8edac6164e4be747830c1e13806981cd2353d26612 not found: ID does not exist"
Sep 17 17:32:58.664638 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.664594    2577 scope.go:117] "RemoveContainer" containerID="6d919b99af026bce8f7a3615eab6507cb67a134028850243375b4ee98143f75c"
Sep 17 17:32:58.664915 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:32:58.664892    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d919b99af026bce8f7a3615eab6507cb67a134028850243375b4ee98143f75c\": container with ID starting with 6d919b99af026bce8f7a3615eab6507cb67a134028850243375b4ee98143f75c not found: ID does not exist" containerID="6d919b99af026bce8f7a3615eab6507cb67a134028850243375b4ee98143f75c"
Sep 17 17:32:58.665068 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.664919    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d919b99af026bce8f7a3615eab6507cb67a134028850243375b4ee98143f75c"} err="failed to get container status \"6d919b99af026bce8f7a3615eab6507cb67a134028850243375b4ee98143f75c\": rpc error: code = NotFound desc = could not find container \"6d919b99af026bce8f7a3615eab6507cb67a134028850243375b4ee98143f75c\": container with ID starting with 6d919b99af026bce8f7a3615eab6507cb67a134028850243375b4ee98143f75c not found: ID does not exist"
Sep 17 17:32:58.666804 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.666784    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-442a34-predictor-668f45cfcc-qdwrg"]
Sep 17 17:32:58.702436 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.702417    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/378254bf-ac71-46fe-8afa-aa28bf90634d-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:32:58.702436 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.702435    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj2xp\" (UniqueName: \"kubernetes.io/projected/378254bf-ac71-46fe-8afa-aa28bf90634d-kube-api-access-sj2xp\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:32:58.702550 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.702445    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/378254bf-ac71-46fe-8afa-aa28bf90634d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:32:58.702550 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.702455    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/378254bf-ac71-46fe-8afa-aa28bf90634d-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:32:58.702550 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:32:58.702469    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/378254bf-ac71-46fe-8afa-aa28bf90634d-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:33:00.447762 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:00.447730    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" path="/var/lib/kubelet/pods/378254bf-ac71-46fe-8afa-aa28bf90634d/volumes"
Sep 17 17:33:03.427716 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:03.427677    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 17:33:13.427126 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:13.427081    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 17:33:23.427495 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:23.427459    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 17:33:31.442325 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:31.442285    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:33:42.440681 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:42.440647    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"]
Sep 17 17:33:42.441059 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:42.440868    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kserve-container" containerID="cri-o://2e5ff8cf3830ef54d9e83b4e7f20c87b4b6ef5ba2a49efd95af057fc6913ebd3" gracePeriod=30
Sep 17 17:33:42.441059 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:42.440902    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kube-rbac-proxy" containerID="cri-o://61756d49c1a30880dedc405ee0b3823b470a2594fdf674510d32c3bdf3c14b4b" gracePeriod=30
Sep 17 17:33:42.846372 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:42.846342    2577 generic.go:356] "Generic (PLEG): container finished" podID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerID="61756d49c1a30880dedc405ee0b3823b470a2594fdf674510d32c3bdf3c14b4b" exitCode=2
Sep 17 17:33:42.846522 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:42.846420    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" event={"ID":"57bd54f1-3087-4bc9-8696-9d8c0c88555a","Type":"ContainerDied","Data":"61756d49c1a30880dedc405ee0b3823b470a2594fdf674510d32c3bdf3c14b4b"}
Sep 17 17:33:43.422500 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:43.422452    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Sep 17 17:33:46.098929 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.098906    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:33:46.141254 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.141231    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kserve-provision-location\") pod \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") "
Sep 17 17:33:46.141367 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.141266    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kube-rbac-proxy-sar-config\") pod \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") "
Sep 17 17:33:46.141367 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.141292    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxdqb\" (UniqueName: \"kubernetes.io/projected/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kube-api-access-sxdqb\") pod \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") "
Sep 17 17:33:46.141367 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.141308    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57bd54f1-3087-4bc9-8696-9d8c0c88555a-proxy-tls\") pod \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") "
Sep 17 17:33:46.141367 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.141322    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57bd54f1-3087-4bc9-8696-9d8c0c88555a-cabundle-cert\") pod \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\" (UID: \"57bd54f1-3087-4bc9-8696-9d8c0c88555a\") "
Sep 17 17:33:46.141627 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.141597    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57bd54f1-3087-4bc9-8696-9d8c0c88555a" (UID: "57bd54f1-3087-4bc9-8696-9d8c0c88555a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:33:46.141703 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.141646    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "57bd54f1-3087-4bc9-8696-9d8c0c88555a" (UID: "57bd54f1-3087-4bc9-8696-9d8c0c88555a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:33:46.141767 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.141726    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57bd54f1-3087-4bc9-8696-9d8c0c88555a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57bd54f1-3087-4bc9-8696-9d8c0c88555a" (UID: "57bd54f1-3087-4bc9-8696-9d8c0c88555a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:33:46.143349 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.143322    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57bd54f1-3087-4bc9-8696-9d8c0c88555a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "57bd54f1-3087-4bc9-8696-9d8c0c88555a" (UID: "57bd54f1-3087-4bc9-8696-9d8c0c88555a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:33:46.143452 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.143404    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kube-api-access-sxdqb" (OuterVolumeSpecName: "kube-api-access-sxdqb") pod "57bd54f1-3087-4bc9-8696-9d8c0c88555a" (UID: "57bd54f1-3087-4bc9-8696-9d8c0c88555a"). InnerVolumeSpecName "kube-api-access-sxdqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:33:46.242043 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.241977    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:33:46.242043 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.242014    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:33:46.242043 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.242026    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxdqb\" (UniqueName: \"kubernetes.io/projected/57bd54f1-3087-4bc9-8696-9d8c0c88555a-kube-api-access-sxdqb\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:33:46.242043 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.242035    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57bd54f1-3087-4bc9-8696-9d8c0c88555a-proxy-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:33:46.242043 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.242043    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57bd54f1-3087-4bc9-8696-9d8c0c88555a-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Sep 17 17:33:46.865201 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.865155    2577 generic.go:356] "Generic (PLEG): container finished" podID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerID="2e5ff8cf3830ef54d9e83b4e7f20c87b4b6ef5ba2a49efd95af057fc6913ebd3" exitCode=0
Sep 17 17:33:46.865386 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.865207    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" event={"ID":"57bd54f1-3087-4bc9-8696-9d8c0c88555a","Type":"ContainerDied","Data":"2e5ff8cf3830ef54d9e83b4e7f20c87b4b6ef5ba2a49efd95af057fc6913ebd3"}
Sep 17 17:33:46.865386 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.865223    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"
Sep 17 17:33:46.865386 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.865270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn" event={"ID":"57bd54f1-3087-4bc9-8696-9d8c0c88555a","Type":"ContainerDied","Data":"303cfcb9e7543479f786f187a6d3a227a4ae1df3aa6e365687b85ddf506decb7"}
Sep 17 17:33:46.865386 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.865306    2577 scope.go:117] "RemoveContainer" containerID="61756d49c1a30880dedc405ee0b3823b470a2594fdf674510d32c3bdf3c14b4b"
Sep 17 17:33:46.873892 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.873871    2577 scope.go:117] "RemoveContainer" containerID="2e5ff8cf3830ef54d9e83b4e7f20c87b4b6ef5ba2a49efd95af057fc6913ebd3"
Sep 17 17:33:46.881393 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.881363    2577 scope.go:117] "RemoveContainer" containerID="5ba8b48d206d86bbb22abeab8f21cb741b8ae270ad2c706a4a43c4e6f742dd5d"
Sep 17 17:33:46.883438 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.883417    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"]
Sep 17 17:33:46.889527 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.889507    2577 scope.go:117] "RemoveContainer" containerID="61756d49c1a30880dedc405ee0b3823b470a2594fdf674510d32c3bdf3c14b4b"
Sep 17 17:33:46.889784 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.889758    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-fqldn"]
Sep 17 17:33:46.889856 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:33:46.889830    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61756d49c1a30880dedc405ee0b3823b470a2594fdf674510d32c3bdf3c14b4b\": container with ID starting with 61756d49c1a30880dedc405ee0b3823b470a2594fdf674510d32c3bdf3c14b4b not found: ID does not exist" containerID="61756d49c1a30880dedc405ee0b3823b470a2594fdf674510d32c3bdf3c14b4b"
Sep 17 17:33:46.889899 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.889852    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61756d49c1a30880dedc405ee0b3823b470a2594fdf674510d32c3bdf3c14b4b"} err="failed to get container status \"61756d49c1a30880dedc405ee0b3823b470a2594fdf674510d32c3bdf3c14b4b\": rpc error: code = NotFound desc = could not find container \"61756d49c1a30880dedc405ee0b3823b470a2594fdf674510d32c3bdf3c14b4b\": container with ID starting with 61756d49c1a30880dedc405ee0b3823b470a2594fdf674510d32c3bdf3c14b4b not found: ID does not exist"
Sep 17 17:33:46.889899 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.889869    2577 scope.go:117] "RemoveContainer" containerID="2e5ff8cf3830ef54d9e83b4e7f20c87b4b6ef5ba2a49efd95af057fc6913ebd3"
Sep 17 17:33:46.890125 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:33:46.890106    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5ff8cf3830ef54d9e83b4e7f20c87b4b6ef5ba2a49efd95af057fc6913ebd3\": container with ID starting with 2e5ff8cf3830ef54d9e83b4e7f20c87b4b6ef5ba2a49efd95af057fc6913ebd3 not found: ID does not exist" containerID="2e5ff8cf3830ef54d9e83b4e7f20c87b4b6ef5ba2a49efd95af057fc6913ebd3"
Sep 17 17:33:46.890249 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.890130    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5ff8cf3830ef54d9e83b4e7f20c87b4b6ef5ba2a49efd95af057fc6913ebd3"} err="failed to get container status \"2e5ff8cf3830ef54d9e83b4e7f20c87b4b6ef5ba2a49efd95af057fc6913ebd3\": rpc error: code = NotFound desc = could not find container \"2e5ff8cf3830ef54d9e83b4e7f20c87b4b6ef5ba2a49efd95af057fc6913ebd3\": container with ID starting with 2e5ff8cf3830ef54d9e83b4e7f20c87b4b6ef5ba2a49efd95af057fc6913ebd3 not found: ID does not exist"
Sep 17 17:33:46.890249 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.890144    2577 scope.go:117] "RemoveContainer" containerID="5ba8b48d206d86bbb22abeab8f21cb741b8ae270ad2c706a4a43c4e6f742dd5d"
Sep 17 17:33:46.890417 ip-10-0-135-124 kubenswrapper[2577]: E0917 17:33:46.890400    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba8b48d206d86bbb22abeab8f21cb741b8ae270ad2c706a4a43c4e6f742dd5d\": container with ID starting with 5ba8b48d206d86bbb22abeab8f21cb741b8ae270ad2c706a4a43c4e6f742dd5d not found: ID does not exist" containerID="5ba8b48d206d86bbb22abeab8f21cb741b8ae270ad2c706a4a43c4e6f742dd5d"
Sep 17 17:33:46.890458 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:46.890421    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba8b48d206d86bbb22abeab8f21cb741b8ae270ad2c706a4a43c4e6f742dd5d"} err="failed to get container status \"5ba8b48d206d86bbb22abeab8f21cb741b8ae270ad2c706a4a43c4e6f742dd5d\": rpc error: code = NotFound desc = could not find container \"5ba8b48d206d86bbb22abeab8f21cb741b8ae270ad2c706a4a43c4e6f742dd5d\": container with ID starting with 5ba8b48d206d86bbb22abeab8f21cb741b8ae270ad2c706a4a43c4e6f742dd5d not found: ID does not exist"
Sep 17 17:33:48.447351 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:33:48.447308    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" path="/var/lib/kubelet/pods/57bd54f1-3087-4bc9-8696-9d8c0c88555a/volumes"
Sep 17 17:34:28.050102 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:28.050061    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9snxm_fd70adfb-9928-407c-b57b-d51617deb1cc/global-pull-secret-syncer/0.log"
Sep 17 17:34:28.236811 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:28.236780    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7dxcc_a8418cea-e387-4ea4-8a0b-c69dccd07ba5/konnectivity-agent/0.log"
Sep 17 17:34:28.322130 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:28.322070    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-124.ec2.internal_2c421660fe0335a8468fb794163e0e9e/haproxy/0.log"
Sep 17 17:34:32.965546 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:32.965512    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-lvqv5_b03c9158-2b71-4a29-a557-823ddc1bd610/monitoring-plugin/0.log"
Sep 17 17:34:33.005179 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:33.005142    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqd7m_32b46210-87b1-43c6-aa0e-ca201d6e8097/node-exporter/0.log"
Sep 17 17:34:33.036716 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:33.036690    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqd7m_32b46210-87b1-43c6-aa0e-ca201d6e8097/kube-rbac-proxy/0.log"
Sep 17 17:34:33.065686 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:33.065668    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqd7m_32b46210-87b1-43c6-aa0e-ca201d6e8097/init-textfile/0.log"
Sep 17 17:34:33.834556 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:33.834532    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57648dc9b4-mbl62_29d15cf0-b377-48e4-99a9-ff34677d7b82/thanos-query/0.log"
Sep 17 17:34:33.865927 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:33.865901    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57648dc9b4-mbl62_29d15cf0-b377-48e4-99a9-ff34677d7b82/kube-rbac-proxy-web/0.log"
Sep 17 17:34:33.889378 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:33.889353    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57648dc9b4-mbl62_29d15cf0-b377-48e4-99a9-ff34677d7b82/kube-rbac-proxy/0.log"
Sep 17 17:34:33.917736 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:33.917714    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57648dc9b4-mbl62_29d15cf0-b377-48e4-99a9-ff34677d7b82/prom-label-proxy/0.log"
Sep 17 17:34:33.942985 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:33.942936    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57648dc9b4-mbl62_29d15cf0-b377-48e4-99a9-ff34677d7b82/kube-rbac-proxy-rules/0.log"
Sep 17 17:34:33.968635 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:33.968611    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57648dc9b4-mbl62_29d15cf0-b377-48e4-99a9-ff34677d7b82/kube-rbac-proxy-metrics/0.log"
Sep 17 17:34:35.552083 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:35.552054    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/2.log"
Sep 17 17:34:35.559999 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:35.559978    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-s9jpc_0b353d50-4511-4e9d-a50d-362d67b3e222/console-operator/3.log"
Sep 17 17:34:36.008252 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008172    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"]
Sep 17 17:34:36.008514 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008500    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kube-rbac-proxy"
Sep 17 17:34:36.008564 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008515    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kube-rbac-proxy"
Sep 17 17:34:36.008564 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008529    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="storage-initializer"
Sep 17 17:34:36.008564 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008535    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="storage-initializer"
Sep 17 17:34:36.008564 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008541    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kube-rbac-proxy"
Sep 17 17:34:36.008564 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008547    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kube-rbac-proxy"
Sep 17 17:34:36.008564 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008556    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="storage-initializer"
Sep 17 17:34:36.008564 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008561    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="storage-initializer"
Sep 17 17:34:36.008784 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008574    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kserve-container"
Sep 17 17:34:36.008784 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008579    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kserve-container"
Sep 17 17:34:36.008784 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008584    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kserve-container"
Sep 17 17:34:36.008784 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008589    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kserve-container"
Sep 17 17:34:36.008784 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008636    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kserve-container"
Sep 17 17:34:36.008784 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008645    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="57bd54f1-3087-4bc9-8696-9d8c0c88555a" containerName="kube-rbac-proxy"
Sep 17 17:34:36.008784 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008651    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kube-rbac-proxy"
Sep 17 17:34:36.008784 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.008657    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="378254bf-ac71-46fe-8afa-aa28bf90634d" containerName="kserve-container"
Sep 17 17:34:36.011541 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.011519    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.014003 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.013980    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrk88\"/\"kube-root-ca.crt\""
Sep 17 17:34:36.014133 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.014041    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lrk88\"/\"default-dockercfg-4l9gt\""
Sep 17 17:34:36.014133 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.014102    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrk88\"/\"openshift-service-ca.crt\""
Sep 17 17:34:36.020712 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.020689    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"]
Sep 17 17:34:36.050393 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.050361    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-dpwv7_de57cca8-b211-4c6d-8691-93914d97d54f/download-server/0.log"
Sep 17 17:34:36.099935 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.099910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12f74e0c-20ee-40cd-b1f3-d55e56a3bc05-sys\") pod \"perf-node-gather-daemonset-vsfjp\" (UID: \"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.100086 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.099942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm742\" (UniqueName: \"kubernetes.io/projected/12f74e0c-20ee-40cd-b1f3-d55e56a3bc05-kube-api-access-gm742\") pod \"perf-node-gather-daemonset-vsfjp\" (UID: \"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.100086 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.099964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/12f74e0c-20ee-40cd-b1f3-d55e56a3bc05-proc\") pod \"perf-node-gather-daemonset-vsfjp\" (UID: \"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.100086 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.099986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/12f74e0c-20ee-40cd-b1f3-d55e56a3bc05-podres\") pod \"perf-node-gather-daemonset-vsfjp\" (UID: \"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.100086 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.100027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12f74e0c-20ee-40cd-b1f3-d55e56a3bc05-lib-modules\") pod \"perf-node-gather-daemonset-vsfjp\" (UID: \"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.200580 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.200545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12f74e0c-20ee-40cd-b1f3-d55e56a3bc05-lib-modules\") pod \"perf-node-gather-daemonset-vsfjp\" (UID: \"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.200728 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.200598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12f74e0c-20ee-40cd-b1f3-d55e56a3bc05-sys\") pod \"perf-node-gather-daemonset-vsfjp\" (UID: \"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.200728 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.200620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm742\" (UniqueName: \"kubernetes.io/projected/12f74e0c-20ee-40cd-b1f3-d55e56a3bc05-kube-api-access-gm742\") pod \"perf-node-gather-daemonset-vsfjp\" (UID: \"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.200728 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.200649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/12f74e0c-20ee-40cd-b1f3-d55e56a3bc05-proc\") pod \"perf-node-gather-daemonset-vsfjp\" (UID: \"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.200728 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.200679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/12f74e0c-20ee-40cd-b1f3-d55e56a3bc05-podres\") pod \"perf-node-gather-daemonset-vsfjp\" (UID: \"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.200728 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.200693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12f74e0c-20ee-40cd-b1f3-d55e56a3bc05-sys\") pod \"perf-node-gather-daemonset-vsfjp\" (UID: \"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.200937 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.200741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12f74e0c-20ee-40cd-b1f3-d55e56a3bc05-lib-modules\") pod \"perf-node-gather-daemonset-vsfjp\" (UID: \"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.200937 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.200759    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/12f74e0c-20ee-40cd-b1f3-d55e56a3bc05-proc\") pod \"perf-node-gather-daemonset-vsfjp\" (UID: \"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.200937 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.200804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/12f74e0c-20ee-40cd-b1f3-d55e56a3bc05-podres\") pod \"perf-node-gather-daemonset-vsfjp\" (UID: \"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.208517 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.208493    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm742\" (UniqueName: \"kubernetes.io/projected/12f74e0c-20ee-40cd-b1f3-d55e56a3bc05-kube-api-access-gm742\") pod \"perf-node-gather-daemonset-vsfjp\" (UID: \"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05\") " pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.321739 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.321716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:36.571553 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.571519    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-cfz55_1e65ad15-c59c-4005-ace7-1122b4be237a/volume-data-source-validator/0.log"
Sep 17 17:34:36.596020 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:36.595826    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"]
Sep 17 17:34:36.598619 ip-10-0-135-124 kubenswrapper[2577]: W0917 17:34:36.598593    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod12f74e0c_20ee_40cd_b1f3_d55e56a3bc05.slice/crio-143148c37798e5f99647d5e6067aef22771e92cd60fc49daa8dd8215da8b67cf WatchSource:0}: Error finding container 143148c37798e5f99647d5e6067aef22771e92cd60fc49daa8dd8215da8b67cf: Status 404 returned error can't find the container with id 143148c37798e5f99647d5e6067aef22771e92cd60fc49daa8dd8215da8b67cf
Sep 17 17:34:37.089066 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:37.089032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp" event={"ID":"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05","Type":"ContainerStarted","Data":"9006da6e2bb9fd3f75f0917249d76e51fa1db105c4851f4fe0cd3043b32ae4c5"}
Sep 17 17:34:37.089237 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:37.089088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp" event={"ID":"12f74e0c-20ee-40cd-b1f3-d55e56a3bc05","Type":"ContainerStarted","Data":"143148c37798e5f99647d5e6067aef22771e92cd60fc49daa8dd8215da8b67cf"}
Sep 17 17:34:37.089237 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:37.089117    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:37.103839 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:37.103775    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp" podStartSLOduration=2.103760823 podStartE2EDuration="2.103760823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:37.103322699 +0000 UTC m=+3195.128231000" watchObservedRunningTime="2026-09-17 17:34:37.103760823 +0000 UTC m=+3195.128669113"
Sep 17 17:34:37.492532 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:37.492448    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kz9dn_d8d1f173-79e9-40f2-adf4-221f6e68df5c/dns/0.log"
Sep 17 17:34:37.511429 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:37.511398    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kz9dn_d8d1f173-79e9-40f2-adf4-221f6e68df5c/kube-rbac-proxy/0.log"
Sep 17 17:34:37.627353 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:37.627325    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fvkfn_c95ed8dc-a8ee-4dc0-a7f2-1facc60e5ab1/dns-node-resolver/0.log"
Sep 17 17:34:38.100992 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:38.100965    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6d9xb_f4c6d895-d5c6-4540-b230-c8d40d3d4205/node-ca/0.log"
Sep 17 17:34:39.234499 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:39.234446    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lsznc_3eea7da6-118b-4a18-9f0c-0e509101fdec/serve-healthcheck-canary/0.log"
Sep 17 17:34:39.643154 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:39.643115    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-678qx_e2fbafb9-cf8e-4b97-8971-c231a952ba1e/insights-operator/0.log"
Sep 17 17:34:39.643938 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:39.643916    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-678qx_e2fbafb9-cf8e-4b97-8971-c231a952ba1e/insights-operator/1.log"
Sep 17 17:34:39.663147 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:39.663127    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4zxsg_d9a0ec35-0982-4a2a-841d-b24cf11b8e37/kube-rbac-proxy/0.log"
Sep 17 17:34:39.682735 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:39.682717    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4zxsg_d9a0ec35-0982-4a2a-841d-b24cf11b8e37/exporter/0.log"
Sep 17 17:34:39.702326 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:39.702306    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4zxsg_d9a0ec35-0982-4a2a-841d-b24cf11b8e37/extractor/0.log"
Sep 17 17:34:42.246937 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:42.246894    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-7mlwz_6211f1f7-b0dc-4d47-bef4-f51028d720ad/manager/0.log"
Sep 17 17:34:43.013703 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:43.013676    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-2qzhg_c45a52e9-4d7d-464e-9ae6-9e73113e5585/seaweedfs/0.log"
Sep 17 17:34:43.103753 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:43.103727    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lrk88/perf-node-gather-daemonset-vsfjp"
Sep 17 17:34:49.268796 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:49.268764    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phjts_2b2d9a82-bf68-49e0-b54a-df73639f138a/kube-multus-additional-cni-plugins/0.log"
Sep 17 17:34:49.292562 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:49.292542    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phjts_2b2d9a82-bf68-49e0-b54a-df73639f138a/egress-router-binary-copy/0.log"
Sep 17 17:34:49.316727 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:49.316697    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phjts_2b2d9a82-bf68-49e0-b54a-df73639f138a/cni-plugins/0.log"
Sep 17 17:34:49.343509 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:49.343486    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phjts_2b2d9a82-bf68-49e0-b54a-df73639f138a/bond-cni-plugin/0.log"
Sep 17 17:34:49.367933 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:49.367916    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phjts_2b2d9a82-bf68-49e0-b54a-df73639f138a/routeoverride-cni/0.log"
Sep 17 17:34:49.392512 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:49.392467    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phjts_2b2d9a82-bf68-49e0-b54a-df73639f138a/whereabouts-cni-bincopy/0.log"
Sep 17 17:34:49.417426 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:49.417405    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phjts_2b2d9a82-bf68-49e0-b54a-df73639f138a/whereabouts-cni/0.log"
Sep 17 17:34:49.464920 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:49.464892    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cv7kh_9c1d6b1e-f3b7-4a15-ae25-eacdcf3b0fbe/kube-multus/0.log"
Sep 17 17:34:49.569480 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:49.569457    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fqmmn_35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a/network-metrics-daemon/0.log"
Sep 17 17:34:49.588794 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:49.588773    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fqmmn_35ef9dbe-6ecf-42dc-9237-9bf1986b4d7a/kube-rbac-proxy/0.log"
Sep 17 17:34:50.433313 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:50.433284    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w4sg_4ec628ab-cd9d-4e35-a77c-3ba182d19487/ovn-controller/0.log"
Sep 17 17:34:50.457574 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:50.457550    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w4sg_4ec628ab-cd9d-4e35-a77c-3ba182d19487/ovn-acl-logging/0.log"
Sep 17 17:34:50.482667 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:50.482645    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w4sg_4ec628ab-cd9d-4e35-a77c-3ba182d19487/kube-rbac-proxy-node/0.log"
Sep 17 17:34:50.507565 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:50.507542    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w4sg_4ec628ab-cd9d-4e35-a77c-3ba182d19487/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 17:34:50.526890 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:50.526868    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w4sg_4ec628ab-cd9d-4e35-a77c-3ba182d19487/northd/0.log"
Sep 17 17:34:50.547504 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:50.547479    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w4sg_4ec628ab-cd9d-4e35-a77c-3ba182d19487/nbdb/0.log"
Sep 17 17:34:50.572254 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:50.572234    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w4sg_4ec628ab-cd9d-4e35-a77c-3ba182d19487/sbdb/0.log"
Sep 17 17:34:50.636942 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:50.636919    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w4sg_4ec628ab-cd9d-4e35-a77c-3ba182d19487/ovnkube-controller/0.log"
Sep 17 17:34:52.052818 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:52.052784    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-sk4pl_af8b4478-4c0f-4076-8327-261955205e93/check-endpoints/0.log"
Sep 17 17:34:52.098849 ip-10-0-135-124 kubenswrapper[2577]: I0917 17:34:52.098829    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-jn44m_2c9aee2e-6b4d-4c04-9a63-d2f81312f45f/network-check-target-container/0.log"